Local Image Features Research Articles

Non-Contact Measurement of Pregnant Sows’ Backfat Thickness Based on a Hybrid CNN-ViT Model

Backfat thickness (BF) is closely related to the service life and reproductive performance of sows. The dynamic monitoring of sows’ BF is a critical part of the production process in large-scale pig farms. This study proposed the application of a hybrid CNN-ViT (Vision Transformer, ViT) model for measuring sows’ BF to address the problems of high measurement intensity caused by the traditional contact measurement of sows’ BF and the low efficiency of existing non-contact models for measuring sows’ BF. The CNN-ViT introduced depth-separable convolution and lightweight self-attention, mainly consisting of a Pre-local Unit (PLU), a Lightweight ViT (LViT) and an Inverted Residual Unit (IRU). This model could extract local and global features of images, making it more suitable for small datasets. The model was tested on 106 pregnant sows with seven randomly divided datasets. The results showed that the CNN-ViT had a Mean Absolute Error (MAE) of 0.83 mm, a Root Mean Square Error (RMSE) of 1.05 mm, a Mean Absolute Percentage Error (MAPE) of 4.87% and a coefficient of determination (R-Square, R2) of 0.74. Compared to LviT-IRU, PLU-IRU and PLU-LviT, the CNN-ViT’s MAE decreased by more than 12%, RMSE decreased by more than 15%, MAPE decreased by more than 15% and R² improved by more than 17%. Compared to the Resnet50 and ViT, the CNN-ViT’s MAE decreased by more than 7%, RMSE decreased by more than 13%, MAPE decreased by more than 7% and R2 improved by more than 15%. The method could better meet the demand for the non-contact automatic measurement of pregnant sows’ BF in actual production and provide technical support for the intelligent management of pregnant sows.

A rotation-invariant corner detector based on the median of subpixelized triangle

Corners are stable local image features that have played a crucial role in various computer vision and image processing tasks. Existing corner detectors generally assume that the distance between every two adjacent pixels is a constant during curvature estimation. However, this assumption is actually invalid, and thus those pixel-based measures of discrete curvature developed and exploited in the existing corner detectors may suffer instability under rotation transformations. To address this fundamental problem, a novel curvature measure is proposed in this paper, which exploits the median of a subpixelized triangle located at the current point to estimate its discrete curvature. It is shown that our proposed curvature measure is invariant under rotation transformations. Based on this novel curvature measure, a new corner detector is further developed. Extensive experimental results show that our proposed corner detector can deliver superior performance over the existing state-of-the-art methods.

AEA-Net:Affinity-supervised entanglement attentive network for person re-identification

Most existing person re-identification algorithms prioritize the extraction of effective and salient local features while neglecting the affinity between local and adjacent features. This phenomenon will cause misidentification when different people have similar local features. To address this issue, we introduce the AEA-Net, which emphasizes the affinity between the local features of a single image. Specifically, three important components are proposed. The affinity-supervised attention module (ASA) centers on adjacent features and utilizes the affinity between global and adjacent features to supervise the learning of attention. The affinity relationship module (AR) focuses on constructing relationship features between local and adjacent features to enhance the closeness between local features. The tangle hybrid loss (THL) makes the final predictions have a distinct weight profile. Extensive experiments quantitatively and qualitatively demonstrate that our method outperforms the state-of-the-art approaches.

Design of a Robust Hybrid Fuzzy Method for Medical Image Fusion

In this modern era, medical image processing is an indispensable part of many applications and practices in the medical domain. The images that are used should meet certain criteria, including having more accurate details and information than each individual image, which can help medical scientists with analysis and treatment. Medical image fusion is among the techniques that offer high-quality images, which are combined from different modalities. Multimodal medical image fusion provides remarkable improvement in the quality of the fused images. In this paper, we describe an image fusion method for magnetic resonance imaging (MRI) and computed tomography (CT) utilizing local features and fuzzy logic methods. The aim of the proposed technique is to create the maximum combination of useful information present in MRI and CT images. Image local features are distinguished and combined with fuzzy logic to calculate weights for each pixel. Simulation outcomes show that the proposed method produces considerably better results compared to cutting-edge techniques. The method is also used to detect and highlight tumorous areas, followed by morphology filters used to eliminate any noise and disturbance.

Implementation of Augmented Reality at Interactive Food Menu Using the Speed Up Robust Features (SURF) Algorithm

— Promotion is an attempt to notify or offer a product or service with the aim of attracting potential customers to buy or consume it, with the promotion, producers or distributors expect an increase in sales figures. In this study researchers used Augmented Reality technology for interactive media promotion of food menus by adding 3D multimedia elements. The method used in this study uses the Natural Feature Tracking method with the Speed Up Robust Features (SURF) algorithm, which detects local features in marker images that are resistant to rotation, scale and blurring. The results showed that the keypoint functions to render 3D objects. Search for keypoints is interrupted due to distance, light intensity and slope of the marker. Test results to see the distance between the camera and the marker as far as 60 cm. Medium light intensity that can detect markers, the average time of object speed can be displayed is 3.026 seconds and the marker slope limit is 30 °. This is because the keypoint readings at the position and time limit from keypoint readings clearly, keypoint readings clearly produce 3D objects can be displayed. This research uses the Android platform as the foundation of this Augmented Reality technology application. So that by displaying 3D food menu items in restaurants it is expected to be a means of promotion to attract consumers.

ESPT: A Self-Supervised Episodic Spatial Pretext Task for Improving Few-Shot Learning

Self-supervised learning (SSL) techniques have recently been integrated into the few-shot learning (FSL) framework and have shown promising results in improving the few-shot image classification performance. However, existing SSL approaches used in FSL typically seek the supervision signals from the global embedding of every single image. Therefore, during the episodic training of FSL, these methods cannot capture and fully utilize the local visual information in image samples and the data structure information of the whole episode, which are beneficial to FSL. To this end, we propose to augment the few-shot learning objective with a novel self-supervised Episodic Spatial Pretext Task (ESPT). Specifically, for each few-shot episode, we generate its corresponding transformed episode by applying a random geometric transformation to all the images in it. Based on these, our ESPT objective is defined as maximizing the local spatial relationship consistency between the original episode and the transformed one. With this definition, the ESPT-augmented FSL objective promotes learning more transferable feature representations that capture the local spatial features of different images and their inter-relational structural information in each input episode, thus enabling the model to generalize better to new categories with only a few samples. Extensive experiments indicate that our ESPT method achieves new state-of-the-art performance for few-shot image classification on three mainstay benchmark datasets. The source code will be available at: https://github.com/Whut-YiRong/ESPT.

Context-aware lightweight remote-sensing image super-resolution network.

In recent years, remote-sensing image super-resolution (RSISR) methods based on convolutional neural networks (CNNs) have achieved significant progress. However, the limited receptive field of the convolutional kernel in CNNs hinders the network's ability to effectively capture long-range features in images, thus limiting further improvements in model performance. Additionally, the deployment of existing RSISR models to terminal devices is challenging due to their high computational complexity and large number of parameters. To address these issues, we propose a Context-Aware Lightweight Super-Resolution Network (CALSRN) for remote-sensing images. The proposed network primarily consists of Context-Aware Transformer Blocks (CATBs), which incorporate a Local Context Extraction Branch (LCEB) and a Global Context Extraction Branch (GCEB) to explore both local and global image features. Furthermore, a Dynamic Weight Generation Branch (DWGB) is designed to generate aggregation weights for global and local features, enabling dynamic adjustment of the aggregation process. Specifically, the GCEB employs a Swin Transformer-based structure to obtain global information, while the LCEB utilizes a CNN-based cross-attention mechanism to extract local information. Ultimately, global and local features are aggregated using the weights acquired from the DWGB, capturing the global and local dependencies of the image and enhancing the quality of super-resolution reconstruction. The experimental results demonstrate that the proposed method is capable of reconstructing high-quality images with fewer parameters and less computational complexity compared with existing methods.

Dynamic Weighting Network for Person Re-Identification.

Recently, hybrid Convolution-Transformer architectures have become popular due to their ability to capture both local and global image features and the advantage of lower computational cost over pure Transformer models. However, directly embedding a Transformer can result in the loss of convolution-based features, particularly fine-grained features. Therefore, using these architectures as the backbone of a re-identification task is not an effective approach. To address this challenge, we propose a feature fusion gate unit that dynamically adjusts the ratio of local and global features. The feature fusion gate unit fuses the convolution and self-attentive branches of the network with dynamic parameters based on the input information. This unit can be integrated into different layers or multiple residual blocks, which will have varying effects on the accuracy of the model. Using feature fusion gate units, we propose a simple and portable model called the dynamic weighting network or DWNet, which supports two backbones, ResNet and OSNet, called DWNet-R and DWNet-O, respectively. DWNet significantly improves re-identification performance over the original baseline, while maintaining reasonable computational consumption and number of parameters. Finally, our DWNet-R achieves an mAP of 87.53%, 79.18%, 50.03%, on the Market1501, DukeMTMC-reID, and MSMT17 datasets. Our DWNet-O achieves an mAP of 86.83%, 78.68%, 55.66%, on the Market1501, DukeMTMC-reID, and MSMT17 datasets.

A Multiscale Local–Global Feature Fusion Method for SAR Image Classification with Bayesian Hyperparameter Optimization Algorithm

In recent years, the advancement of deep learning technology has led to excellent performance in synthetic aperture radar (SAR) automatic target recognition (ATR) technology. However, due to the interference of speckle noise, the task of classifying SAR images remains challenging. To address this issue, a multi-scale local–global feature fusion network (MFN) integrating a convolution neural network (CNN) and a transformer network was proposed in this study. The proposed network comprises three branches: a CovNeXt-SimAM branch, a Swin Transformer branch, and a multi-scale feature fusion branch. The CovNeXt-SimAM branch extracts local texture detail features of the SAR images at different scales. By incorporating the SimAM attention mechanism to the CNN block, the feature extraction capability of the model was enhanced from the perspective of spatial and channel attention. Additionally, the Swin Transformer branch was employed to extract SAR image global semantic information at different scales. Finally, the multi-scale feature fusion branch was used to fuse local features and global semantic information. Moreover, to overcome the problem of poor accuracy and inefficiency of the model due to empirically determined model hyperparameters, the Bayesian hyperparameter optimization algorithm was used to determine the optimal model hyperparameters. The model proposed in this study achieved average recognition accuracies of 99.26% and 94.27% for SAR vehicle targets under standard operating conditions (SOCs) and extended operating conditions (EOCs), respectively, on the MSTAR dataset. Compared with the baseline model, the recognition accuracy has been improved by 12.74% and 25.26%, respectively. The results demonstrated that Bayes-MFN reduces the inter-class distance of the SAR images, resulting in more compact classification features and less interference from speckle noise. Compared with other mainstream models, the Bayes-MFN model exhibited the best classification performance.

Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network With Multiscale Feature Fusion

Reconstructing visual stimulus from human brain activity measured with functional magnetic resonance imaging (fMRI) is a challenging decoding task for revealing the visual system. Recent deep learning approaches commonly neglect the relationship between hierarchical image features and different regions of the visual cortex, and fail to use global and local image features in reconstructing visual stimulus. To address these issues, in this paper, a novel neural decoding framework is proposed by using a dual attention hierarchical latent generative network with multi-scale feature fusion (DA-HLGN-MSFF) method. Specifically, the fMRI data is firstly encoded to hierarchical features of our image encoder network which employs a multi-kernel convolution block to extract the multi-scale spatial information of images. In order to reconstruct the perceived images and further improve the performance of our generator network, a dual attention block based on channel-spatial attention mechanism is then proposed to exploit the inter-channel relationships and spatial long-range dependencies of features. Moreover, a multi-scale feature fusion block is finally adopted to aggregate the global and local information of features at different scales and synthesize the final reconstructed images in the generator network. Competitive experimental results on two public fMRI datasets demonstrate that our method is able to achieve promising reconstructing performance compared with the state-of-the-art methods. The codes of our proposed DA-HLGN-MSFF method will be open access on https://github.com/ljbuaa/HLDAGN.

Perception-Oriented U-Shaped Transformer Network for 360-Degree No-Reference Image Quality Assessment

Generally, 360-degree images have absolute senses of reality and three-dimensionality, providing a wide range of immersive interactions. Due to the novel rendering and display technology of 360-degree images, they have more complex perceptual characteristics than other images. It is challenging to perform comprehensive image quality assessment (IQA) learning by simply stacking multichannel neural network architectures for pre/postprocessing, compression, and rendering tasks. To thoroughly learn the global and local features in 360-degree images, reduce the complexity of multichannel neural network models and simplify the training process, this paper proposes a joint architecture with user perception and an efficient transformer dedicated to 360-degree no-reference (NR) IQA. The input of the proposed method is a 360-degree cube map projection (CMP) image. Furthermore, the proposed 360-degree NRIQA method includes a saliency map-based non-overlapping self-attention selection module and a U-shaped transformer (U-former)-based feature extraction module to account for perceptual region importance and projection distortion. The transformer-based architecture and the weighted average technique are jointly utilized for predicting local perceptual quality. Experimental results obtained on widely used databases show that the proposed model outperforms other state-of-the-art methods in NR 360-degree image quality evaluation cases. Furthermore, a cross-database evaluation and an ablation study also demonstrate the inherent robustness and generalization ability of the proposed model.

High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet

To solve the problem of poor quality in ghost imaging via sparsity constraints (GISC) multispectral image reconstruction with correlation operations and compressed sensing algorithms under low sampling rate detection conditions, we propose an end-to-end deep-learning-based method. Based on the U-Net, Res2Net-SE-Conv is employed instead of convolutional blocks to extract local and global image features at a more fine-grained level while adaptively adjusting the channel feature response. The two-dimensional contextual transformer is constructed to fully use contextual correlation information to enhance the effectiveness of feature representations. We employ the two-dimensional contextual transformer in the decoder part, dubbed CoT-Unet, to reconstruct the desired 3D cube. The results show that compared with U-Net, TSA-Net based on spatial-spectral self-attention, the PSNR of reconstructed images by the CoT-Unet is improved by 5 dB and 3 dB, respectively, SSIM is improved by 0.23 and 0.07, and SAM is decreased by 0.06 and 0.58. Compared with conventional algorithms such as DGI and CS, our method significantly improves the quality of reconstructed images. Furthermore, the comparison results at 10%, 20%, and 30% sampling rates show that our approach has the best quality in reconstructing GISC multispectral images at low sampling rates.

Semi-Dense Feature Matching with Transformers and its Applications in Multiple-View Geometry.

We present a novel method for local image feature matching. Instead of performing image feature detection, description, and matching sequentially, we propose to first establish pixel-wise dense matches at a coarse level and later refine the good matches at a fine level. In contrast to dense methods that use a cost volume to search correspondences, we use self and cross attention layers in Transformer to obtain feature descriptors that are conditioned on both images. The global receptive field provided by Transformer enables our method to produce dense matches in low-texture areas, where feature detectors usually struggle to produce repeatable interest points. The experiments on indoor and outdoor datasets show that LoFTR outperforms state-of-the-art methods by a large margin. We further adapt LoFTR to modern SfM systems and illustrate its application in multiple-view geometry. The proposed method demonstrates superior performance in Image Matching Challenge 2021 and ranks first on two public benchmarks of visual localization among the published methods. The code is available at https://zju3dv.github.io/loftr.

An improved checkerboard detection algorithm based on adaptive filters

Checkerboard corner extraction is a crucial step in camera calibration. However, most existing algorithms are not good enough if the lens distortion is too large. This study aims to propose a checkerboard corner detection algorithm based on adaptive filters to address this problem. First, adaptive filters based on local image features are used to generate response maps of checkerboard images. Next, a non-max suppression and a scoring system are applied for further screening. Finally, the whole checkerboard structure is restored via inertia growth. The algorithm proposed is subject to rigorous experimental validation using synthetic and real images. Compared with several state of the art methods, our algorithm has the best performance.

CICHMKG: a large-scale and comprehensive Chinese intangible cultural heritage multimodal knowledge graph

Intangible Cultural Heritage (ICH) witnesses human creativity and wisdom in long histories, composed of a variety of immaterial manifestations. The rapid development of digital technologies accelerates the record of ICH, generating a sheer number of heterogenous data but in a state of fragmentation. To resolve that, existing studies mainly adopt approaches of knowledge graphs (KGs) which can provide rich knowledge representation. However, most KGs are text-based and text-derived, and incapable to give related images and empower downstream multimodal tasks, which is also unbeneficial for the public to establish the visual perception and comprehend ICH completely especially when they do not have the related ICH knowledge. Hence, aimed at that, we propose to, taking the Chinese nation-level ICH list as an example, construct a large-scale and comprehensive Multimodal Knowledge Graph (CICHMKG) combining text and image entities from multiple data sources and give a practical construction framework. Additionally, in this paper, to select representative images for ICH entities, we propose a method composed of the denoising algorithm (CNIFA) and a series of criteria, utilizing global and local visual features of images and textual features of captions. Extensive empirical experiments demonstrate its effectiveness. Lastly, we construct the CICHMKG, consisting of 1,774,005 triples, and visualize it to facilitate the interactions and help the public dive into ICH deeply.

HIGF-Net: Hierarchical information-guided fusion network for polyp segmentation based on transformer and convolution feature learning

Polyp segmentation plays a role in image analysis during colonoscopy screening, thus improving the diagnostic efficiency of early colorectal cancer. However, due to the variable shape and size characteristics of polyps, small difference between lesion area and background, and interference of image acquisition conditions, existing segmentation methods have the phenomenon of missing polyp and rough boundary division. To overcome the above challenges, we propose a multi-level fusion network called HIGF-Net, which uses hierarchical guidance strategy to aggregate rich information to produce reliable segmentation results. Specifically, our HIGF-Net excavates deep global semantic information and shallow local spatial features of images together with Transformer encoder and CNN encoder. Then, Double-stream structure is used to transmit polyp shape properties between feature layers at different depths. The module calibrates the position and shape of polyps in different sizes to improve the model's efficient use of the rich polyp features. In addition, Separate Refinement module refines the polyp profile in the uncertain region to highlight the difference between the polyp and the background. Finally, in order to adapt to diverse collection environments, Hierarchical Pyramid Fusion module merges the features of multiple layers with different representational capabilities. We evaluate the learning and generalization abilities of HIGF-Net on five datasets using six evaluation metrics, including Kvasir-SEG, CVC-ClinicDB, ETIS, CVC-300, and CVC-ColonDB. Experimental results show that the proposed model is effective in polyp feature mining and lesion identification, and its segmentation performance is better than ten excellent models.

New Fusion Network with Dual-Branch Encoder and Triple-Branch Decoder for Remote Sensing Image Change Detection

Deep learning plays a highly essential role in the domain of remote sensing change detection (CD) due to its high efficiency. From some existing methods, we can observe that the fusion of information at each scale is quite vital for the accuracy of the CD results, especially for the common problems of pseudo-change and the difficult detection of change edges in the CD task. With this in mind, we propose a New Fusion network with Dual-branch Encoder and Triple-branch Decoder (DETDNet) that follows a codec structure as a whole, where the encoder adopts a siamese Res2Net-50 structure to extract the local features of the bitemporal images. As for the decoder in previous works, they usually employed a single branch, and this approach only preserved the fusion features of the encoder’s bitemporal images. Distinguished from these approaches, we adopt the triple-branch architecture in the decoder for the first time. The triple-branch structure preserves not only the dual-branch features from the encoder in the left and right branches, respectively, to learn the effective and powerful individual features of each temporal image but also the fusion features from the encoder in the middle branch. The middle branch utilizes triple-branch aggregation (TA) to realize the feature interaction of the three branches in the decoder, which enhances the integrated features and provides abundant and supplementary bitemporal feature information to improve the CD performance. The triple-branch architecture of the decoder ensures that the respective features of the bitemporal images as well as their fused features are preserved, making the feature extraction more integrated. In addition, the three branches employ a multiscale feature extraction module (MFE) per layer to extract multiscale contextual information and enhance the feature representation capability of the CD. We conducted comparison experiments on the BCDD, LEVIR-CD, and SYSU-CD datasets, which were created in New Zealand, the USA, and Hong Kong, respectively. The data were preprocessed to contain 7434, 10,192, and 20,000 image pairs, respectively. The experimental results show that DETDNet achieves F1 scores of 92.7%, 90.99%, and 81.13%, respectively, which shows better results compared to some recent works, which means that the model is more robust. In addition, the lower FP and FN indicate lower error and misdetection rates. Moreover, from the analysis of the experimental results, compared with some existing methods, the problem of pseudo-changes and the difficulty of detecting small change areas is better solved.

A CNN-Based Layer-Adaptive GCPs Extraction Method for TIR Remote Sensing Images

Ground Control Points (GCPs) are of great significance for applications involving the registration and fusion of heterologous remote sensing images (RSIs). However, utilizing low-level information rather than deep features, traditional methods based on intensity and local image features turn out to be unsuitable for heterologous RSIs because of the large nonlinear radiation difference (NRD), inconsistent resolutions, and geometric distortions. Additionally, the limitations of current heterologous datasets and existing deep-learning-based methods make it difficult to obtain enough precision GCPs from different kinds of heterologous RSIs, especially for thermal infrared (TIR) images that present low spatial resolution and poor contrast. In this paper, to address the problems above, we propose a convolutional neural network-based (CNN-based) layer-adaptive GCPs extraction method for TIR RSIs. Particularly, the constructed feature extraction network is comprised of basic and layer-adaptive modules. The former is used to achieve the coarse extraction, and the latter is designed to obtain high-accuracy GCPs by adaptively updating the layers in the module to capture the fine communal homogenous features of the heterologous RSIs until the GCP precision meets the preset threshold. Experimental results evaluated on TIR images of SDGSAT-1 TIS and near infrared (NIR), short wave infrared (SWIR), and panchromatic (PAN) images of Landsat-8 OLI show that the matching root-mean-square error (RMSE) of TIS images with SWIR and NIR images could reach 0.8 pixels and an even much higher accuracy of 0.1 pixels could be reached between TIS and PAN images, which performs better than those of the traditional methods, such as SIFT, RIFT, and the CNN-based method like D2-Net.

An End-to-End Framework Based on Vision-Language Fusion for Remote Sensing Cross-Modal Text-Image Retrieval

Remote sensing cross-modal text-image retrieval (RSCTIR) has recently attracted extensive attention due to its advantages of fast extraction of remote sensing image information and flexible human–computer interaction. Traditional RSCTIR methods mainly focus on improving the performance of uni-modal feature extraction separately, and most rely on pre-trained object detectors to obtain better local feature representation, which not only lack multi-modal interaction information, but also cause the training gap between the pre-trained object detector and the retrieval task. In this paper, we propose an end-to-end RSCTIR framework based on vision-language fusion (EnVLF) consisting of two uni-modal (vision and language) encoders and a muti-modal encoder which can be optimized by multitask training. Specifically, to achieve an end-to-end training process, we introduce a vision transformer module for image local features instead of a pre-trained object detector. By semantic alignment of visual and text features, the vision transformer module achieves the same performance as pre-trained object detectors for image local features. In addition, the trained multi-modal encoder can improve the top-one and top-five ranking performances after retrieval processing. Experiments on common RSICD and RSITMD datasets demonstrate that our EnVLF can obtain state-of-the-art retrieval performance.

MLP-based classification of COVID-19 and skin diseases

Recent years have witnessed a growing interest in neural network-based medical image classification methods, which have demonstrated remarkable performance in this field. Typically, convolutional neural network (CNN) architectures have been commonly employed to extract local features. However, the transformer, a newly emerged architecture, has gained popularity due to its ability to explore the relevance of remote elements in an image through a self-attention mechanism. Despite this, it is crucial to establish not only local connectivity but also remote relationships between lesion features and capture the overall image structure to improve image classification accuracy. Therefore, to tackle the aforementioned issues, this paper proposes a network based on multilayer perceptrons (MLPs) that can learn the local features of medical images on the one hand and capture the overall feature information in both spatial and channel dimensions on the other hand, thus utilizing image features effectively. This paper has been extensively validated on COVID19-CT dataset and ISIC 2018 dataset, and the results show that the method in this paper is more competitive and has higher performance in medical image classification compared with existing methods. This shows that the use of MLP to capture image features and establish connections between lesions is expected to provide novel ideas for medical image classification tasks in the future.