Local Image Features Research Articles

MTC-CSNet: Marrying Transformer and Convolution for Image Compressed Sensing.

Image compressed sensing (ICS) has been extensively applied in various imaging domains due to its capability to sample and reconstruct images at subNyquist sampling rates. The current predominant approaches in ICS, specifically pure convolutional networks (ConvNets)-based ICS methods, have demonstrated their effectiveness in capturing local features for image recovery. Simultaneously, the Transformer architecture has gained significant attention due to its capability to model global correlations among image features. Motivated by these insights, we propose a novel hybrid network for ICS, named MTC-CSNet, which effectively combines the strengths of both ConvNets and Transformer architectures in capturing local and global image features to achieve high-quality image recovery. Particularly, MTC-CSNet is a dual-path framework that consists of a ConvNets-based recovery branch and a Transformer-based recovery branch. Along the ConvNets-based recovery branch, we design a lightweight scheme to capture the local features in natural images. Meanwhile, we implement a Transformer-based recovery branch to iteratively model the global dependencies among image patches. Ultimately, the ConvNets-based and Transformer-based recovery branches collaborate through a bridging unit, facilitating the adaptive transmission and fusion of informative features for image reconstruction. Extensive experimental results demonstrate that our proposed MTC-CSNet surpasses the state-of-the-art methods on various public datasets. The code and models are publicly available at MTC-CSNet.

Medical X-Ray Image Enhancement Using Global Contrast-Limited Adaptive Histogram Equalization

In medical imaging, accurate diagnosis heavily relies on effective image enhancement techniques, particularly for X-ray images. Existing methods often suffer from various challenges such as sacrificing global image characteristics over local image characteristics or vice versa. In this paper, we present a novel approach, called G-CLAHE (Global-Contrast Limited Adaptive Histogram Equalization), which perfectly suits medical imaging with a focus on X-rays. This method adapts from Global Histogram Equalization (GHE) and Contrast Limited Adaptive Histogram Equalization (CLAHE) to take both advantages and avoid weakness to preserve local and global characteristics. Experimental results show that it can significantly improve current state-of-the-art algorithms to effectively address their limitations and enhance the contrast and quality of X-ray images for diagnostic accuracy.

Saliency and edge features-guided end-to-end network for salient object detection

The rapid development of the Vision Transformer backbones has enabled the capture of feature information with global dependencies, leading to excellent performance in salient object detection tasks. However, it fails to adequately emphasize fine local features of edges, resulting in coarse and blurry edges in the final output. Therefore, this paper proposes a set prediction method for salient object detection. The approach enables the model to simultaneously consider both saliency (salient object) and edge features, achieving an end-to-end edge feature fusion. There is no need to design multiple complex branch structures and multiple training as with other methods. The model integrates random edge neighborhood sampling to enhance the recognition accuracy of local edge features in images. This approach addresses the weak perception of local features by Transformers and the issue encountered during actual training, where edge pixels typically occupy a much smaller proportion of the image compared to the background, resulting in insufficient learning of edge features by the model. The proposed end-to-end model in this paper fuses multiple features, including edge and salient objects. They are extracted within a unified framework while simultaneously outputting edge and salient object maps. Experimental results on six public datasets show that the proposed method significantly improves model performance benchmarks for detecting salient objects.

Dual-branch Transformer for semi-supervised medical image segmentation.

In recent years, the use of deep learning for medical image segmentation has become a popular trend, but its development also faces some challenges. Firstly, due to the specialized nature of medical data, precise annotation is time-consuming and labor-intensive. Training neural networks effectively with limited labeled data is a significant challenge in medical image analysis. Secondly, convolutional neural networks commonly used for medical image segmentation research often focus on local features in images. However, the recognition of complex anatomical structures or irregular lesions often requires the assistance of both local and global information, which has led to a bottleneck in its development. Addressing these two issues, in this paper, we propose a novel network architecture. We integrate a shift window mechanism to learn more comprehensive semantic information and employ a semi-supervised learning strategy by incorporating a flexible amount of unlabeled data. Specifically, a typical U-shaped encoder-decoder structure is applied to obtain rich feature maps. Each encoder is designed as a dual-branch structure, containing Swin modules equipped with windows of different size to capture features of multiple scales. To effectively utilize unlabeled data, a level set function is introduced to establish consistency between the function regression and pixel classification. We conducted experiments on the COVID-19 CT dataset and DRIVE dataset and compared our approach with various semi-supervised and fully supervised learning models. On the COVID-19 CT dataset, we achieved a segmentation accuracy of up to 74.56%. Our segmentation accuracy on the DRIVE dataset was 79.79%. The results demonstrate the outstanding performance of our method on several commonly used evaluation metrics. The high segmentation accuracy of our model demonstrates that utilizing Swin modules with different window sizes can enhance the feature extraction capability of the model, and the level set function can enable semi-supervised models to more effectively utilize unlabeled data. This provides meaningful insights for the application of deep learning in medical image segmentation. Our code will be released once the manuscript is accepted for publication.

Global and local complementary multi-path feature fusion network for the classification of crop remote sensing images

Abstract The accuracy and efficiency of crop distribution information extraction are pivotal in ensuring global food security. In long-time-series optical satellite data, most existing methods focus on extracting spatial features using Convolutional Neural Networks (CNNs), which do not adequately mine and model the spatial-temporal information. The development of the attention mechanism allows for the extraction of global features in remote-sensing images of long temporal sequences. To extract global attentional features with complementary features in crop remote sensing images, we propose a Global and Local Complementary Multi-path Feature Fusion Network (GLMP), which is capable of extracting global features from remote sensing images of long temporal sequence, that enhances the local characteristics of crop images derived from CNNs, thus obtaining more effective multi-scale complementary features. This extraction of features enhances the comprehension of crop images, thereby boosting the performance of associated tasks. Within GLMP, we introduce two pivotal modules: the Hybrid Attention and Convolutional Paths Module (HACM) and the Multi-path Feature Fusion Module (MPFM). These modules synergistically converge multi-path features, yielding more discriminative feature information. Experimental results on the ZueriCrop dataset show that the proposed GLMP technique is effective; it performs promisingly having a total accuracy of 90.2% and an F1 value of 62.5%. Furthermore, the ablation study verifies the substantial improvement in classification accuracy for remote sensing crop images of long-time series in nature, specifically attributed to the HACM and MPFM modules.

Chinese traditional painting style automatic classification based on dual-channel feature fusion with multi-attention mechanism

Existing classification models for traditional Chinese paintings mostly ignore shallow detail features, which leads to the imprecise classification of styles. To address the above problems, this paper proposes a Chinese traditional painting style automatic classification model based on dual-channel feature fusion with multi-attention mechanism. First, the spatial attention mechanism is introduced to enhance the Swin-Transformer framework to obtain the salient features of Chinese ancient painting images. Second, a dual-channel attention mechanism is constructed to extract global semantic features and local features of Chinese ancient painting images. Finally, the extracted features are fused and categorized based on the softmax classifier. To verify the feasibility and validity of the proposed model, this paper performs simulations on the Chinese painting dataset and compares it with existing algorithms.The average classification accuracy of the proposed model is 90.6[Formula: see text], with an improvement of 3.14[Formula: see text], which is better than the existing model in both visual effects and objective data comparisons.

Scaling Painting Style Transfer

AbstractNeural style transfer (NST) is a deep learning technique that produces an unprecedentedly rich style transfer from a style image to a content image. It is particularly impressive when it comes to transferring style from a painting to an image. NST was originally achieved by solving an optimization problem to match the global statistics of the style image while preserving the local geometric features of the content image. The two main drawbacks of this original approach is that it is computationally expensive and that the resolution of the output images is limited by high GPU memory requirements. Many solutions have been proposed to both accelerate NST and produce images with larger size. However, our investigation shows that these accelerated methods all compromise the quality of the produced images in the context of painting style transfer. Indeed, transferring the style of a painting is a complex task involving features at different scales, from the color palette and compositional style to the fine brushstrokes and texture of the canvas. This paper provides a solution to solve the original global optimization for ultra‐high resolution (UHR) images, enabling multiscale NST at unprecedented image sizes. This is achieved by spatially localizing the computation of each forward and backward passes through the VGG network. Extensive qualitative and quantitative comparisons, as well as a perceptual study, show that our method produces style transfer of unmatched quality for such high‐resolution painting styles. By a careful comparison, we show that state‐of‐the‐art fast methods are still prone to artifacts, thus suggesting that fast painting style transfer remains an open problem.

Global contrast-masked autoencoders are powerful pathological representation learners

Using digital pathology slide scanning technology, artificial intelligence algorithms, particularly deep learning, have achieved significant results in the field of computational pathology. Compared to other medical images, pathology images are more difficult to annotate, and thus, there is an extreme lack of available datasets for conducting supervised learning to train robust deep learning models. In this paper, we introduce a self-supervised learning (SSL) model, the Global Contrast-masked Autoencoder (GCMAE), designed to train encoders to capture both local and global features of pathological images and significantly enhance the performance of transfer learning across datasets. Our study demonstrates the capability of the GCMAE to learn transferable representations through extensive experiments on three distinct disease-specific hematoxylin and eosin (H&E)-stained pathology datasets: Camelyon16, NCT-CRC, and BreakHis. Moreover, we propose an effective automated pathology diagnosis process based on the GCMAE for clinical applications. The source code of this paper is publicly available at https://github.com/StarUniversus/gcmae.

A High-Precision Identification Method for Maize Leaf Diseases and Pests Based on LFMNet under Complex Backgrounds.

Maize, as one of the most important crops in the world, faces severe challenges from various diseases and pests. The timely and accurate identification of maize leaf diseases and pests is of great significance for ensuring agricultural production. Currently, the identification of maize leaf diseases and pests faces two key challenges: (1) In the actual process of identifying leaf diseases and pests, complex backgrounds can interfere with the identification effect. (2) The subtle features of diseases and pests are difficult to accurately extract. To address these challenges, this study proposes a maize leaf disease and pest identification model called LFMNet. Firstly, the localized multi-scale inverted residual convolutional block (LMSB) is proposed to perform preliminary down-sampling on the image, preserving important feature information for the subsequent extraction of fine disease and pest features in the model structure. Then, the feature localization bottleneck (FLB) is proposed to improve the model's ability to focus on and locate disease and pest characteristics and to reduce interference from complex backgrounds. Subsequently, the multi-hop local-feature fusion architecture (MLFFA) is proposed, which effectively addresses the problem of extracting subtle features by enhancing the extraction and fusion of global and local disease and pest features in images. After training and testing on a dataset containing 19,451 images of maize leaf diseases and pests, the LFMNet model demonstrated excellent performance, with an average identification accuracy of 95.68%, a precision of 95.91%, a recall of 95.78%, and an F1 score of 95.83%. Compared to existing models, it exhibits significant advantages, offering robust technical support for the precise identification of maize diseases and pests.

Brain Cognition-Inspired Dual-Pathway CNN Architecture for Image Classification.

Inspired by the global-local information processing mechanism in the human visual system, we propose a novel convolutional neural network (CNN) architecture named cognition-inspired network (CogNet) that consists of a global pathway, a local pathway, and a top-down modulator. We first use a common CNN block to form the local pathway that aims to extract fine local features of the input image. Then, we use a transformer encoder to form the global pathway to capture global structural and contextual information among local parts in the input image. Finally, we construct the learnable top-down modulator where fine local features of the local pathway are modulated by global representations of the global pathway. For ease of use, we encapsulate the dual-pathway computation and modulation process into a building block, called the global-local block (GL block), and a CogNet of any depth can be constructed by stacking a necessary number of GL blocks one after another. Extensive experimental evaluations have revealed that the proposed CogNets have achieved the state-of-the-art performance accuracies on all the six benchmark datasets and are very effective for overcoming the "texture bias" and the "semantic confusion" problems faced by many CNN models.

Radiological image analysis using effective channel extension and fusion network based on COVID CT images

Attention mechanisms demonstrate significant capabilities in improving the performance of COVID CT image classification networks. Existing strategies often employ global pooling to acquire feature vectors, overlooking local feature information. Complex attention modules are then applied to enhance network performance, inevitably leading to increased model complexity. To address these issues, this paper introduces an Effective Channel Expansion and Fusion (ECEF) module specifically designed to enhance the performance of medical COVID CT image classification. The module represents a simple yet effective convolutional neural network module. In the channel direction of the given feature map, the module performs segmentation, conducts global pooling operations separately on the segmented and overall feature maps to obtain corresponding attention vectors, and merges them to produce high-dimensional attention vectors. Channel attention feature vectors are derived along the two pooling operation directions and dynamically fused. The resulting attention vectors are multiplied by the input feature map for adaptive feature refinement. Through these operations, the ECEF module adeptly captures local features in COVID CT images, enhancing the classification performance of medical COVID CT images. As a lightweight and versatile module, ECEF seamlessly integrates into any CNN architecture with negligible overhead. It can undergo end-to-end training alongside basic CNNs. ResNet is selected as the backbone network for empirical studies conducted on COVID CT images, including the COVID-CT dataset. Experimental results indicate that, compared to current advanced channel attention mechanisms, the ECEF module exhibits superior performance in COVID CT image classification tasks. The successful application of the ECEF module establishes a foundation for its widespread use in areas such as COVID CT medical image segmentation and object detection.

A Visible and Synthetic Aperture Radar Image Fusion Algorithm Based on a Transformer and a Convolutional Neural Network

For visible and Synthetic Aperture Radar (SAR) image fusion, this paper proposes a visible and SAR image fusion algorithm (TCFuse) based on a Transformer and a Convolutional Neural Network (CNN). Firstly, in this paper, the Restormer Block is used to extract cross-modal shallow features. Then, we introduce an improved Transformer–CNN Feature Extractor (TCFE) with a two-branch residual structure. This includes a Transformer branch that introduces the Lite Transformer (LT) and DropKey for extracting global features and a CNN branch that introduces the Convolutional Block Attention Module (CBAM) for extracting local features. Finally, the fused image is output based on global features extracted by the Transformer branch and local features extracted by the CNN branch. The experiments show that the algorithm proposed in this paper can effectively achieve the extraction and fusion of global and local features of visible and SAR images, so that high-quality visible and SAR fusion images can be obtained.

Research on the Application of VR Technology based on Human Posture Recognition in Online Physical Education

Offline and current online physical education classes are characterized by low learning efficiency and poor teaching quality. To solve this problem, the study combines a dynamic time regularization algorithm and local image feature action recognition with virtual technology, which is applied to online teaching to improve teaching quality. Experimental results indicate that the overall model accuracy reaches more than 94%, and in the public dataset MSR, the model accuracy reaches 91.4%. In the UTK dataset, the accuracy reaches 85.8%. When the accuracy of pose recognition is compared, the research model is superior to the two traditional models. The validation results against KNN and SVM are 62% and 71% as well as 79% and 84%, respectively, and the experimental results of the research model have improved by 9% and 5%. The research technique achieves an overall fit of more than 90% for different parts in the analysis of motion capture, which is superior to the traditional online instruction. The overall teaching satisfaction reaches more than 93.5% in the comparison of online physical education courses with offline courses. In summary, the optimized and improved posture recognition system is better for human motion capture, and the applied virtual technology can effectively improve the learning effect and teaching quality of online physical education courses.

A unified 2D medical image segmentation network (SegmentNet) through distance-awareness and local feature extraction

In addressing the challenges of medical image segmentation, particularly the elusiveness of global context and limitations in leveraging both global and local context simultaneously, we present SegmentNet as a solution. Our approach involves a step-by-step implementation within the reconstructed UNet architecture, tailored to enhance segmentation performance across diverse medical imaging modalities. The first step involves the integration of multi-focus Distance-Aware Mechanisms (DaMs) within skip connections and between successive layers of the encoder in SegmentNet. This strategic placement focuses on extracting unrelated features, ensuring comprehensive consideration of global context. Following this, Local Feature Extractor Blocks (LFEBs) are introduced at the base of the network. Equipped with depthwise separable operations, standard convolutions, smoothed ReLU, and normalization transform, LFEBs target the capture of specific local image features ensuring that features overlooked by DaMs are appropriately considered. These extracted features are then passed on to the decoder portion of SegmentNet, facilitating enhanced prediction of masks thus, optimizing segmentation performance. Evaluated across diverse datasets, including Breast Ultrasound Images (BUSI), Chest X-ray images (CXRI), and Diabetic Retinal Fundus Images (DRFI), SegmentNet excels. The segmentation evaluation results in terms of accuracy, Jaccard, and specificity are respectively recorded for BUSI, CXRI, and DRFI to be (93.88 %, 98.96 %, and 99.17 %), (99.28 %, 99.58 %, and 99.83 %), and (95.77 %, 95.95 %, and 99.94 %). Thus, showing that the incorporation of DaMs and LFEBs in SegmentNet emerges as a robust solution demonstrating precise 2D medical image segmentation across various modalities. This advancement holds significant potential for diverse clinical applications, promising improved patient care.

Chinese image captioning with fusion encoder and visual keyword search

AbstractAutomatic generation of image captions is essentially a cross‐modal conversion from image to text. Owing to the differences in linguistic characteristics between Chinese and English, quite a few Chinese image captioning methods have recently been proposed. Nevertheless, the existing Chinese image captioning models usually lack attention to local details of images or tend to produce general descriptions. To address these challenges, a Chinese image captioning method is proposed that incorporates fusion encoder, visual keyword search, and reinforcement learning. The fusion encoder can simultaneously extract local and global features of the input image to enrich the semantic information in the decoding stage, visual keyword search can pursue potential visual words associated with the image content, and the reinforcement learning mechanism can optimize the evaluation metric CIDEr at sentence level to promote the lexical diversity of image description. The results of extensive experiments demonstrate that the proposed model outperforms the state‐of‐the‐art models and delivers expressive and informative Chinese image captions.

Plane-wave medical image reconstruction based on dynamic Criss-Cross attention and multi-scale convolution.

Plane-wave imaging is widely employed in medical imaging due to its ultra-fast imaging speed. However, the image quality is compromised. Existing techniques to enhance image quality tend to sacrifice the imaging frame rate. The study aims to reconstruct high-quality plane-wave images while maintaining the imaging frame rate. The proposed method utilizes a U-Net-based generator incorporating a multi-scale convolution module in the encoder to extract information at different levels. Additionally, a Dynamic Criss-Cross Attention (DCCA) mechanism is proposed in the decoder of the U-Net-based generator to extract both local and global features of plane-wave images while avoiding interference caused by irrelevant regions. In the reconstruction of point targets, the experimental images achieved a reduction in Full Width at Half Maximum (FWHM) of 0.0499 mm, compared to the Coherent Plane-Wave Compounding (CPWC) method using 75-beam plane waves. For the reconstruction of cyst targets, the simulated image achieved a 3.78% improvement in Contrast Ratio (CR) compared to CPWC. The proposed model effectively addresses the issue of unclear lesion sites in plane-wave images.

ICC-BiFormer: A Deep-Learning Model for Near-Earth Asteroid Detection via Image Compression and Local Feature Extraction

Detecting near-Earth asteroids (NEAs) is crucial for research in solar system and planetary science. In recent year, deep-learning methods have almost dominated the task. Since NEAs represent only one-thousandth of the pixels in images, we proposed an ICC-BiFormer model that includes an image compression and contrast enhancement block and a BiFormer model to capture local features in input images, which is different from previous models based on Convolutional Neural Network (CNN). Furthermore, we utilize a larger input size of the model, which corresponds to the side length of the input image matrix, and design a cropping algorithm to prevent NEAs from being truncated and better divide NEAs and satellites. We apply our ICC-BiFormer model into a dataset of approximately 20,000 streak and 40,000 non-streak images to train a binary classification model. The ICC-BiFormer achieves 99.88% accuracy, which is superior to existing models. Focusing on local features has been proven effective in detecting NEAs.

A Hybrid Learning-Architecture for Improved Brain Tumor Recognition

The accurate classification of brain tumors is an important step for early intervention. Artificial intelligence (AI)-based diagnostic systems have been utilized in recent years to help automate the process and provide more objective and faster diagnosis. This work introduces an enhanced AI-based architecture for improved brain tumor classification. We introduce a hybrid architecture that integrates vision transformer (ViT) and deep neural networks to create an ensemble classifier, resulting in a more robust brain tumor classification framework. The analysis pipeline begins with preprocessing and data normalization, followed by extracting three types of MRI-derived information-rich features. The latter included higher-order texture and structural feature sets to harness the spatial interactions between image intensities, which were derived using Haralick features and local binary patterns. Additionally, local deeper features of the brain images are extracted using an optimized convolutional neural networks (CNN) architecture. Finally, ViT-derived features are also integrated due to their ability to handle dependencies across larger distances while being less sensitive to data augmentation. The extracted features are then weighted, fused, and fed to a machine learning classifier for the final classification of brain MRIs. The proposed weighted ensemble architecture has been evaluated on publicly available and locally collected brain MRIs of four classes using various metrics. The results showed that leveraging the benefits of individual components of the proposed architecture leads to improved performance using ablation studies.

Texture Classification Method Based on Local Enhancement and Non-local Median Patterns

Local binary pattern (LBP) serves as a highly powerful method for texture classification. LBP and its variant methods are extensively applied across various domains of image processing.In increasingly complex imaging environments, LBP faces two issues: (1) Loss of detail information during feature extraction. (2) Sensitivity to noise. To mitigate these issues, this paper proposes a method called Local Enhancement and Non-local Median Pattern (LENMP). It consists of two operators: Local Adaptive Contrast Enhancement Pattern (LEP) and Non-local Median Binary Pattern (NMBP). The LEP operator captures the sign and magnitude details of local image characteristics, while the NMBP operator captures the global information of image features. First, the image is processed using the threshold ACE (thACE) algorithm to enhance the contrast of high-frequency information in the image, extracting image contour edge information. Then, median extraction is performed separately on the two operators to capture larger spatial texture detail information. Finally, comparative experiments are conducted on multiple datasets (Outex, CUReT, Brodatz, KTH-TIPS, and Kylberg) with representative texture classification methods. The results indicate that our proposed LENMP texture classification method exhibits good classification performance and remains competitive compared to the latest descriptors.

SkinSwinViT: A Lightweight Transformer-Based Method for Multiclass Skin Lesion Classification with Enhanced Generalization Capabilities

In recent decades, skin cancer has emerged as a significant global health concern, demanding timely detection and effective therapeutic interventions. Automated image classification via computational algorithms holds substantial promise in significantly improving the efficacy of clinical diagnoses. This study is committed to mitigating the challenge of diagnostic accuracy in the classification of multiclass skin lesions. This endeavor is inherently formidable owing to the resemblances among various lesions and the constraints associated with extracting precise global and local image features within diverse dimensional spaces using conventional convolutional neural network methodologies. Consequently, this study introduces the SkinSwinViT methodology for skin lesion classification, a pioneering model grounded in the Swin Transformer framework featuring a global attention mechanism. Leveraging the inherent cross-window attention mechanism within the Swin Transformer architecture, the model adeptly captures local features and interdependencies within skin lesion images while additionally incorporating a global self-attention mechanism to discern overarching features and contextual information effectively. The evaluation of the model’s performance involved the ISIC2018 challenge dataset. Furthermore, data augmentation techniques augmented training dataset size and enhanced model performance. Experimental results highlight the superiority of the SkinSwinViT method, achieving notable metrics of accuracy, recall, precision, specificity, and F1 score at 97.88%, 97.55%, 97.83%, 99.36%, and 97.79%, respectively.