Local Image Regions Research Articles

Structural Repetition Detector: multi-scale quantitative mapping of molecular complexes through microscopy.

From molecules to organelles, cells exhibit recurring structural motifs across multiple scales. Understanding these structures provides insights into their functional roles. While super-resolution microscopy can visualise such patterns, manual detection in large datasets is challenging and biased. We present the Structural Repetition Detector (SReD), an unsupervised computational framework that identifies repetitive biological structures by exploiting local texture repetition. SReD formulates structure detection as a similarity-matching problem between local image regions. It detects recurring patterns without prior knowledge or constraints on the imaging modality. We demonstrate SReD's capabilities on various fluorescence microscopy images. Quantitative analyses of three datasets highlight SReD's utility: estimating the periodicity of spectrin rings in neurons, detecting HIV-1 viral assembly, and evaluating microtubule dynamics modulated by EB3. Our open-source ImageJ and Fiji plugin enables unbiased analysis of repetitive structures across imaging modalities in diverse biological contexts.

FeatWalk: Enhancing Few-Shot Classification through Local View Leveraging

Few-shot learning is a challenging task due to the limited availability of training samples. Recent few-shot learning studies with meta-learning and simple transfer learning methods have achieved promising performance. However, the feature extractor pre-trained with the upstream dataset may neglect the extraction of certain features which could be crucial for downstream tasks. In this study, inspired by the process of human learning in few-shot tasks, where humans not only observe the whole image (`global view') but also attend to various local image regions (`local view') for comprehensive understanding of detailed features, we propose a simple yet effective few-shot learning method called FeatWalk which can utilize the complementary nature of global and local views, therefore providing an intuitive and effective solution to the problem of insufficient local information extraction from the pre-trained feature extractor. Our method can be easily and flexibly combined with various existing methods, further enhancing few-shot learning performance. Extensive experiments on multiple benchmark datasets consistently demonstrate the effectiveness and versatility of our method.The source code is available at https://github.com/exceefind/FeatWalk.

Global Plus Local Jointly Regularized Support Vector Data Description for Novelty Detection.

In many practice application, the cost for acquiring abnormal data is quite expensive, thus the one-class classification (OCC) problem attracts great attention. As one of the solutions, support vector data description (SVDD) gains a continuous focus in outlier detection since it is based on the data description. For the sphere obtained by SVDD, both the center and the volume (or radius) strongly depend on the support vectors, while the support vectors are sensitive to the tradeoff parameter C. Hence, how to select this parameter is a rather challenging problem. In order to address this problem, we define several distance metrics relative to the image region in Gaussian kernel space. With the distance metrics, two probability densities relative to the global region and the local region are designed, respectively. Then, the information quantity and the information entropy are developed for regularizing the tradeoff parameter. This novel SVDD is called global plus local jointly regularized support vector data description (GL-SVDD), in which both the global region information and the local image region information jointly penalize the images as possible outliers. Finally, we use the UCI dataset and the hyperspectral data of cherry fruit to evaluate the performance of several OCC approaches. Experimental results show that GL-SVDD is encouraging.

LFR-GAN: Local Feature Refinement based Generative Adversarial Network for Text-to-Image Generation

Text-to-image generation aims to generate images from text descriptions. Its main challenge lies in two aspects: (1) Semantic consistency, i.e., the generated images should be semantically consistent with the input text; and (2) Visual reality, i.e., the generated images should look like real images. To ensure text-image consistency, existing works mainly learn to establish the cross-modal representations via a text encoder and image encoder. However, due to the limited representation capability of the fixed-length embeddings and the flexibility of the free-form text descriptions, the learned text-to-image model is incapable of maintaining the semantic consistency between image local regions and fine-grained descriptions. As a result, the generated images sometimes miss some fine-grained attributes of the generated object, such as the color or shape of a part of the object. To address this issue, this paper proposes a Local Feature Refinement Based Generative Adversarial Network (LFR-GAN) , which first divides the text into some independent fine-grained attributes and generates an initial image, then refines the image details based on these attributes. The main contributions are three-fold: (1) An attribute modeling approach is proposed to model the fine-grained text descriptions by mapping them into representations of independent attributes, which provides more fine-grained details for image generation. (2) A local feature refinement approach is proposed to enable the generated image to form a complete reflection of the fine-grained attributes contained in the text description. (3) A multi-stage generation approach is proposed to realize the fine-grained manipulation of complex images progressively, which aims to improve the performance of the refinement and generate photo-realistic images. Extensive experiments on the CUB and Oxford102 datasets show the effectiveness of our LFR-GAN approach in both text-to-image generation and text-guided image manipulation tasks. Our LFR-GAN approach shows superior performance compared to the state-of-the-art methods. The codes will be released at https://github.com/PKU-ICST-MIPL/LFR-GAN_TOMM2023 .

Vision Transformer with hierarchical structure and windows shifting for person re-identification.

Extracting rich feature representations is a key challenge in person re-identification (Re-ID) tasks. However, traditional Convolutional Neural Networks (CNN) based methods could ignore a part of information when processing local regions of person images, which leads to incomplete feature extraction. To this end, this paper proposes a person Re-ID method based on vision Transformer with hierarchical structure and window shifting. When extracting person image features, the hierarchical Transformer model is constructed by introducing the hierarchical construction method commonly used in CNN. Then, considering the importance of local information of person images for complete feature extraction, the self-attention calculation is performed by shifting within the window region. Finally, experiments on three standard datasets demonstrate the effectiveness and superiority of the proposed method.

A collaborative gated attention network for fine-grained visual classification

Fine-grained image classification aims at subdividing large coarse-grained categories into finer-grained subcategories. Most existing fine-grained research methods use a single attention mechanism or multiple sub-networks to zoom in and find distinguishable local feature regions. These models seldom explore the intrinsic connections between cross-layer features with similar semantic features. This tends to show erratic performance in images with complex backgrounds. To this end, we propose a feature-semantic fusion module to enhance the diversity of global feature information. Second, we employ cross-layer spatial attention and channel attention modules, which can accurately locate local key regions of images. Finally, we propose a cross-gate attention module that can find rich discriminative features from key object regions of images to guide the final classification. Experiments show that the proposed model performs well on three datasets: CUB-200-2011, Stanford cars, and FGVC aircraft.

Local energy-based image region segmentation using Legendre polynomials

Segmentation of images is a critical pre-processing step in image processing. The objective of segmentation is to divide an image into two homogeneous segments based on similar features. Recently, energy-based segmentation methods, such as the level set algorithm, have exhibited remarkable performance in segmenting various types of image data. However, such methods fail to produce satisfactory results on textured images, leading to inaccurate foreground and background segmentation. To address this issue, the proposed method utilizes a feature space based on image local energy, which is independent of image intensity. This feature space provides a more robust representation of texture in the image. By taking advantage of a unique feature space, images can be represented with a sum of Legendre polynomials. The reproduced image is injected into the L2S algorithm. The level set function is optimized from the average difference of Legendre polynomial coefficients to provide the best representation of the image. Consequently, the proposed method results in better segmentation accuracy, with an average increase of 19.33 compared to existing state-of-the-art methods.

C3DBed: Facial micro-expression recognition with three-dimensional convolutional neural network embedding in transformer model

Facial micro-expression is often used for emotional recognition of people in a high-risk or pressure scene, which may reflect genuine emotions due to the low intensity of facial action units. Current methods focus on locating regions with emotional changes and cropping these regions for local feature extraction. However, these methods may lead to the problem of information redundancy caused by overlapping cropped regions. This paper proposes a novel three-dimensional convolutional neural network embedding in the transformer model (C3DBed). This model learns the attention weight of each local region of the micro-expression image, thereby perceiving the detail changes of the facial image and extracting robust local detail features. Solve the problem of model complexity and information redundancy caused by low-intensity local area positioning of facial muscle movement. The experiment results demonstrated that the proposed C3DBed model achieved competitive performance with accuracy rates of 78.04%, 77.64%, and 75.73% on SMIC, CASME II, and SAMM datasets, respectively.

Deep Semantic-Visual Alignment for zero-shot remote sensing image scene classification

Deep neural networks have achieved promising progress in remote sensing (RS) image classification, for which the training process requires abundant samples for each class. However, it is time-consuming and unrealistic to annotate labels for each RS category, given the fact that the RS target database is increasing dynamically. Zero-shot learning (ZSL) allows for identifying novel classes that are not seen during training, which provides a promising solution for the aforementioned problem. However, previous ZSL models mainly depend on manually-labeled attributes or word embeddings extracted from language models to transfer knowledge from seen classes to novel classes. Those class embeddings may not be visually detectable and the annotation process is time-consuming and labor-intensive. Besides, pioneer ZSL models use convolutional neural networks pre-trained on ImageNet, which focus on the main objects appearing in each image, neglecting the background context that also matters in RS scene classification. To address the above problems, we propose to collect visually detectable attributes automatically. We predict attributes for each class by depicting the semantic-visual similarity between attributes and images. In this way, the attribute annotation process is accomplished by machine instead of human as in other methods. Moreover, we propose a Deep Semantic-Visual Alignment (DSVA) that take advantage of the self-attention mechanism in the transformer to associate local image regions together, integrating the background context information for prediction. The DSVA model further utilizes the attribute attention maps to focus on the informative image regions that are essential for knowledge transfer in ZSL, and maps the visual images into attribute space to perform ZSL classification. With extensive experiments, we show that our model outperforms other state-of-the-art models by a large margin on a challenging large-scale RS scene classification benchmark. Moreover, we qualitatively verify that the attributes annotated by our network are both class discriminative and semantic related, which benefits the zero-shot knowledge transfer.

Centralized Feature Pyramid for Object Detection.

The visual feature pyramid has shown its superiority in both effectiveness and efficiency in a variety of applications. However, current methods overly focus on inter-layer feature interactions while disregarding the importance of intra-layer feature regulation. Despite some attempts to learn a compact intra-layer feature representation with the use of attention mechanisms or vision transformers, they overlook the crucial corner regions that are essential for dense prediction tasks. To address this problem, we propose a Centralized Feature Pyramid (CFP) network for object detection, which is based on a globally explicit centralized feature regulation. Specifically, we first propose a spatial explicit visual center scheme, where a lightweight MLP is used to capture the globally long-range dependencies, and a parallel learnable visual center mechanism is used to capture the local corner regions of the input images. Based on this, we then propose a globally centralized regulation for the commonly-used feature pyramid in a top-down fashion, where the explicit visual center information obtained from the deepest intra-layer feature is used to regulate frontal shallow features. Compared to the existing feature pyramids, CFP not only has the ability to capture the global long-range dependencies but also efficiently obtain an all-round yet discriminative feature representation. Experimental results on the challenging MS-COCO validate that our proposed CFP can achieve consistent performance gains on the state-of-the-art YOLOv5 and YOLOX object detection baselines.

MaskFaceGAN: High Resolution Face Editing with Masked GAN Latent Code Optimization.

Face editing represents a popular research topic within the computer vision and image processing communities. While significant progress has been made recently in this area, existing solutions: (i) are still largely focused on low-resolution images, (ii) often generate editing results with visual artefacts, or (iii) lack fine-grained control over the editing procedure and alter multiple (entangled) attributes simultaneously, when trying to generate the desired facial semantics. In this paper, we aim to address these issues through a novel editing approach, called MaskFaceGAN that focuses on local attribute editing. The proposed approach is based on an optimization procedure that directly optimizes the latent code of a pre-trained (state-of-the-art) Generative Adversarial Network (i.e., StyleGAN2) with respect to several constraints that ensure: (i) preservation of relevant image content, (ii) generation of the targeted facial attributes, and (iii) spatially-selective treatment of local image regions. The constraints are enforced with the help of an (differentiable) attribute classifier and face parser that provide the necessary reference information for the optimization procedure. MaskFaceGAN is evaluated in extensive experiments on the FRGC, SiblingsDB-HQf, and XM2VTS datasets and in comparison with several state-of-the-art techniques from the literature. Our experimental results show that the proposed approach is able to edit face images with respect to several local facial attributes with unprecedented image quality and at high-resolutions (1024×1024), while exhibiting considerably less problems with attribute entanglement than competing solutions. The source code is publicly available from: https://github.com/MartinPernus/MaskFaceGAN.

Facial Expression Recognition Using Dual Path Feature Fusion and Stacked Attention

Facial Expression Recognition (FER) can achieve an understanding of the emotional changes of a specific target group. The relatively small dataset related to facial expression recognition and the lack of a high accuracy of expression recognition are both a challenge for researchers. In recent years, with the rapid development of computer technology, especially the great progress of deep learning, more and more convolutional neural networks have been developed for FER research. Most of the convolutional neural performances are not good enough when dealing with the problems of overfitting from too-small datasets and noise, due to expression-independent intra-class differences. In this paper, we propose a Dual Path Stacked Attention Network (DPSAN) to better cope with the above challenges. Firstly, the features of key regions in faces are extracted using segmentation, and irrelevant regions are ignored, which effectively suppresses intra-class differences. Secondly, by providing the global image and segmented local image regions as training data for the integrated dual path model, the overfitting problem of the deep network due to a lack of data can be effectively mitigated. Finally, this paper also designs a stacked attention module to weight the fused feature maps according to the importance of each part for expression recognition. For the cropping scheme, this paper chooses to adopt a cropping method based on the fixed four regions of the face image, to segment out the key image regions and to ignore the irrelevant regions, so as to improve the efficiency of the algorithm computation. The experimental results on the public datasets, CK+ and FERPLUS, demonstrate the effectiveness of DPSAN, and its accuracy reaches the level of current state-of-the-art methods on both CK+ and FERPLUS, with 93.2% and 87.63% accuracy on the CK+ dataset and FERPLUS dataset, respectively.

Cross-view panorama image synthesis with progressive attention GANs

Despite the significant progress of conditional image generation, it remains difficult to synthesize a ground-view panorama image from a top-view aerial image. Among the core challenges are the vast differences in image appearance and resolution between aerial images and panorama images, and the limited aside information available for top-to-ground viewpoint transformation. To address these challenges, we propose a new Progressive Attention Generative Adversarial Network (PAGAN) with two novel components: a multistage progressive generation framework and a cross-stage attention module. In the first stage, an aerial image is fed into a U-Net-like network to generate one local region of the panorama image and its corresponding segmentation map. Then, the synthetic panorama image region is extended and refined through the following generation stages with our proposed cross-stage attention module that passes semantic information forward stage-by-stage. In each of the successive generation stages, the synthetic panorama image and segmentation map are separately fed into an image discriminator and a segmentation discriminator to compute both later real and fake, as well as feature alignment score maps for discrimination. The model is trained with a novel orientation-aware data augmentation strategy based on the geometric relation between aerial and panorama images. Extensive experimental results on two cross-view datasets show that PAGAN generates high-quality panorama images with more convincing details than state-of-the-art methods.

Learning deep feature correspondence for unsupervised anomaly detection and segmentation

Developing machine learning models that can detect and localize the unexpected or anomalous structures within images is very important for numerous computer vision tasks, such as the defect inspection of manufactured products. However, it is challenging especially when there are few or even no anomalous image samples available. In this paper, we propose an unsupervised mechanism, i.e. deep feature correspondence (DFC), which can be effectively leveraged to detect and segment out the anomalies in images solely with the prior knowledge from anomaly-free samples. We develop our DFC in an asymmetric dual network framework that consists of a generic feature extraction network and an elaborated feature estimation network, and detect the possible anomalies within images by modeling and evaluating the associated deep feature correspondence between the two dual network branches. Furthermore, to improve the robustness of the DFC and further boost the detection performance, we specifically propose a self-feature enhancement (SFE) strategy and a multi-context residual learning (MCRL) network module. Extensive experiments have been carried out to validate the effectiveness of our DFC and the proposed SFE and MCRL. Our approach is very effective for detecting and segmenting the anomalies that appear in confined local regions of images, especially the industrial anomalies. It advances the state-of-the-art performances on the benchmark dataset – MVTec AD. Besides, when applied to a real industrial inspection scene, it outperforms the comparatives by a large margin.

Deep Learning-Based Identification of Maize Leaf Diseases Is Improved by an Attention Mechanism: Self-Attention.

Maize leaf diseases significantly reduce maize yield; therefore, monitoring and identifying the diseases during the growing season are crucial. Some of the current studies are based on images with simple backgrounds, and the realistic field settings are full of background noise, making this task challenging. We collected low-cost red, green, and blue (RGB) images from our experimental fields and public dataset, and they contain a total of four categories, namely, southern corn leaf blight (SCLB), gray leaf spot (GLS), southern corn rust (SR), and healthy (H). This article proposes a model different from convolutional neural networks (CNNs) based on transformer and self-attention. It represents visual information of local regions of images by tokens, calculates the correlation (called attention) of information between local regions with an attention mechanism, and finally integrates global information to make the classification. The results show that our model achieves the best performance compared to five mainstream CNNs at a meager computational cost, and the attention mechanism plays an extremely important role. The disease lesions information was effectively emphasized, and the background noise was suppressed. The proposed model is more suitable for fine-grained maize leaf disease identification in a complex background, and we demonstrated this idea from three perspectives, namely, theoretical, experimental, and visualization.

A multi-scale semantic attention representation for multi-label image recognition with graph networks

Multi-label image recognition is a basic and challenging task in computer vision and multimedia fields. Graph Convolutional Networks (GCNs) are often used to learn the multi-label semantic features and multi-label dependency. Although the label semantic features in GCNs can learn the global image visual representation well, they are rarely used on the local image regions. Therefore, we try to use GCNs to learn global and local features at the same time, and make a balance between them. In this paper, we give a multi-scale semantic attention model MS-SGA-GCN including three main modules (i.e., MS, SGA and GCN) for multi-label image recognition. The Multi-Scale module (MS) utilizes feature maps of different sizes to obtain global features and have strong generalization capabilities. Semantic Guide Attention module (SGA) applies the label embeddings learned by GCNs to guide the generation of the cross-modality class-specific attention maps, which can discover the locations of semantically related regions for each label. Experiments show that our model on two datasets MS-COCO and PASCAL VOC2007 separately achieves the classification accuracy by 83.4% and 94.2%, which has a competitive advantage over other mainstream models.

Learning Tone Curves for Local Image Enhancement

Image enhancement methods can be formulated as global transformations, local transformations, pixel-wise processing, or a mixture of these operations. Global transformations are limited in enhancing local image regions. Existing local and pixel-wise methods mitigate this issue, but give rise to the additional challenge of limited interpretability. Bridging the gap between global and local methods, we propose a local tone mapping network (LTMNet) that learns a grid of tone curves to locally enhance an image. Tone curves are commonly used by photo-editing software and offer an intuitive representation to photographers, facilitating subsequent customization of the image. Tone curves are also widely used in image signal processors (ISPs), making our method easy to deploy on cameras. Because existing datasets contain image enhancement and photofinishing beyond global and local tone mapping, we also propose a new dataset representative of local tone mapping—the LTM dataset. We evaluate our method on this new dataset as well as MIT-Adobe and HDR+ datasets. We show that the proposed LTMNet outperforms existing methods in local tone mapping while achieving competitive performance modeling additional photofinishing. Furthermore, we show that our method can be assistive in user-interactive photo-editing tools. Our code, model, and data will be released publicly.

Tone mapping high dynamic range images based on region-adaptive self-supervised deep learning

This paper presents a region-adaptive self-supervised deep learning (RASSDL) technique for high dynamic range (HDR) image tone mapping. The RASSDL tone mapping operator (TMO) is a convolutional neural network (CNN) trained on local image regions that can seamlessly tone map images of arbitrary sizes. The training of RASSDL TMO is through the design of a self-supervising target that automatically adapts to the local image regions based on their information contents. The self-supervising target is designed to ensure the tone-mapped output achieves a balance between preserving the relative contrast of the original scene and the visibilities of the fine details to achieve faithful reproduction of the HDR scene. Distinguishing from many existing TMOs that require manual tuning of parameters, RASSDL is parameter-free and completely automatic. Experimental results demonstrate that RASSDL TMO can achieve state-of-the-art performance in terms of preserving overall contrasts, revealing fine details, and being free from visual artifacts.

Parameter Estimation of Poisson-Gaussian Signal-Dependent Noise from Single Image of CMOS/CCD Image Sensor Using Local Binary Cyclic Jumping.

Since signal-dependent noise in a local weak texture region of a noisy image is approximated as additive noise, the corresponding noise parameters can be estimated from a given set of weakly textured image blocks. As a result, the meticulous selection of weakly textured image blocks plays a decisive role to estimate the noise parameters accurately. The existing methods consider the finite directions of the texture of image blocks or directly use the average value of an image block to select the weakly textured image block, which can result in errors. To overcome the drawbacks of the existing methods, this paper proposes a novel noise parameter estimation method using local binary cyclic jumping to aid in the selection of these weakly textured image blocks. The texture intensity of the image block is first defined by the cumulative average of the LBCJ information in the eight neighborhoods around the pixel, and, subsequently, the threshold is set for selecting weakly textured image blocks through texture intensity distribution of the image blocks and inverse binomial cumulative function. The experimental results reveal that the proposed method outperforms the existing alternative algorithms by 23% and 22% for the evaluative measures of MSE (a) and MSE (b), respectively.

Search for identical regions in images using invariant moments

One of the ways to describe objects on images is to identify some of their characteristic points or points of attention. Areas of neighborhoods of attention points are described by descriptors (lots of signs) in such way that they can be identified and compared. These signs are used to search for identical points in other images. The article investigates and establishes the possibility of searching for arbitrary local image regions by descriptors constructed with using invariant moments. A feature of the proposed method is that the calculation of the invariant moments of local areas is carried out with using the integral representation of the geometric moments of the image. Integral representation is a matrix with the same size as the image. The elements of the matrix is the sums of the geometric moments of individual pixels, which are located above and to the left with respect to the coordinates of this element. The number of matrices depends on the order of the geometric moments. For moments up to the second order (inclusively), there will be six such matrices. Calculation of one of six geometric moments of an arbitrary rectangular area of the image comes down up to 3 operations such as summation or subtraction of elements of the corresponding matrix located in the corners of this area. The invariant moments are calculated on base of six geometric moments. The search is performed by scanning the image coordinate grid with a window of a given size. In this case, the invariant moments and additional parameters are calculated and compared with similar parameters of the neighborhoods of the reference point of different size (taking into account the possible change in the image scale). The best option is selected according to a given condition. Almost all mass operations of the procedures for calculating the parameters of standards and searching of identical points make it possible explicitly perform parallel computations in the SIMD mode. As a result, the integral representation of geometric moments and the possibility of using parallel computations at all stages will significantly speed up the calculations and allow you to get good indicators of the search efficiency for identical points and the speed of work