Multiscale Image Representation Research Articles

Complete Region of Interest for Unconstrained Palmprint Recognition.

Unconstrained palmprint images have shown great potential for recognition applications due to their lower restrictions regarding hand poses and backgrounds during contactless image acquisition. However, they face two challenges: 1) unclear palm contours and finger-valley points of unconstrained palmprint images make it difficult to locate landmarks to crop the palmprint region of interest (ROI); and 2) large intra-class diversities of unconstrained palmprint images hinder the learning of intra-class-invariant palmprint features. In this paper, we propose to directly extract the complete palmprint region as the ROI (CROI) using the detection-style CenterNet without requiring the detection of any landmarks, and large intra-class diversities may occur. To address this, we further propose a palmprint feature alignment and learning hybrid network (PalmALNet) for unconstrained palmprint recognition. Specifically, we first exploit and align the multi-scale shallow representation of unconstrained palmprint images via deformable convolution and alignment-aware supervision, such that the pixel gaps of the intra-class palmprint CROIs can be minimized in shallow feature space. Then, we develop multiple triple-attention learning modules by integrating spatial, channel, and self-attention operations into convolution to adaptively learn and highlight the latent identity-invariant palmprint information, enhancing the overall discriminative power of the palmprint features. Extensive experimental results on four challenging palmprint databases demonstrate the promising effectiveness of both the proposed PalmALNet and CROI for unconstrained palmprint recognition.

Enhancing medical image object detection with collaborative multi-agent deep Q-networks and multi-scale representation

Object detection holds a crucial role in medical diagnostics. Tasks like organ segmentation and malignancy diagnosis typically necessitate preliminary localization of corresponding anatomical structures. Precise positioning ensures that only pertinent regions require processing, leading to a potential reduction in computational and storage demands. Conventional image detection approaches necessitate numerous candidate boxes, resulting in redundant computations. Developing techniques capable of accurately detecting medical image objects without reliance on candidate boxes holds substantial practical significance. This paper introduces a 2D method for detecting medical image objects, which leverages multi-agent deep Q-network reinforcement learning and a multi-scale image representation. The method constructs a collaborative environment for multiple agents. These agents individually govern the upper-right corner and lower-left corner positions of the object detection frame, progressively converging toward the actual endpoint through iterative interactions. To expedite the detection process, a multi-scale image representation technique is employed. This method segments the process into three scales. Initially, within the coarse-scale space, the agent approximates the region containing the true endpoint, subsequently executing oscillatory movements. Progressively, it refines its approach within the fine-scale space, advancing toward the genuine endpoint with smaller iterative steps. The detection results demonstrate that collaborative detection among agents yields a 2.45%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} higher intersection over union compared to non-collaborative detection. Agents exhibit varying step sizes and fields of view in different scale spaces, leading to a reduction in detection time by 0.12 s compared to single-scale comparison. Experimental outcomes demonstrate the superiority of the medical image target detection method proposed in this study over prevailing mainstream detection algorithms.

Multi-Scale Efficient Graph-Transformer for Whole Slide Image Classification.

The multi-scale information among the whole slide images (WSIs) is essential for cancer diagnosis. Although the existing multi-scale vision Transformer has shown its effectiveness for learning multi-scale image representation, it still cannot work well on the gigapixel WSIs due to their extremely large image sizes. To this end, we propose a novel Multi-scale Efficient Graph-Transformer (MEGT) framework for WSI classification. The key idea of MEGT is to adopt two independent efficient Graph-based Transformer (EGT) branches to process the low-resolution and high-resolution patch embeddings (i.e., tokens in a Transformer) of WSIs, respectively, and then fuse these tokens via a multi-scale feature fusion module (MFFM). Specifically, we design an EGT to efficiently learn the local-global information of patch tokens, which integrates the graph representation into Transformer to capture spatial-related information of WSIs. Meanwhile, we propose a novel MFFM to alleviate the semantic gap among different resolution patches during feature fusion, which creates a non-patch token for each branch as an agent to exchange information with another branch by cross-attention mechanism. In addition, to expedite network training, a new token pruning module is developed in EGT to reduce the redundant tokens. Extensive experiments on both TCGA-RCC and CAMELYON16 datasets demonstrate the effectiveness of the proposed MEGT.

Research on Image Classification and Key Technologies Based on 3D Feature Extraction Algorithm

Image classification and recognition has a very wide range of applications in computer vision, which involves many fields, such as image retrieval, image analysis, and robot positioning. Especially with the rise of brain science and cognitive science research, as well as the increasing diversification of imaging means, three-dimensional image data mainly based on magnetic resonance image plays an increasingly important role in image classification and recognition, especially in medical image classification and recognition. However, due to the high dimensional characteristics of human magnetic resonance images, human readability is reduced. Therefore, classification and recognition of 3-dimensional images is still a challenge. In order to better extract local features from images and effectively use their spatial information, this paper improved the “feature bag” and “spatial pyramid matching” algorithms on the basis of 3D feature extraction algorithm and proposed an image classification framework based on 3D feature extraction algorithm. Firstly, the multiresolution “3D spatial pyramid” algorithm, the multiscale image segmentation and image representation method, and the SVM classifier and feature fusion method are described. Secondly, the gender information contained in the magnetic resonance images is classified and recognized on the three databases selected in the experiment. Experimental results show that this method can effectively utilize the spatial information of three-dimensional images and achieve satisfactory results in the classification and recognition of human magnetic resonance images.

On Franklin wavelets

The Franklin wavelet (piecewise linear wavelet) is a wavelet function [Formula: see text] with dilation factor 2 which is continuous on [Formula: see text], linear on [Formula: see text] and [Formula: see text] for all [Formula: see text]; which is constructed from a multiresolution analysis(MRA) with a scaling function [Formula: see text] which is continuous on [Formula: see text], linear on [Formula: see text] and [Formula: see text] for every [Formula: see text]. Wavelets with composite dilation factor have the ability to produce long and narrow window functions, with varied orientations, well-suited to applications of image processing; also it is more effective and flexible for the construction of multiscale image representation. This motivates us to consider the following problem to construct some composite dilation Franklin wavelets. Given a natural number [Formula: see text], we ask, is there a continuous [Formula: see text] that is linear on [Formula: see text] and [Formula: see text], for [Formula: see text], [Formula: see text] and for all [Formula: see text], which generates an MRA with dilation factor [Formula: see text] to construct an orthonormal [Formula: see text]- wavelet, i.e. a wavelet with dilation factor [Formula: see text]? For [Formula: see text] and [Formula: see text] we have proved that [Formula: see text] generates an MRA with dilation factor [Formula: see text] only if [Formula: see text]. Next for [Formula: see text] and [Formula: see text], we have proved that, if [Formula: see text] generates an MRA with dilation factor [Formula: see text], then [Formula: see text]. Also, for [Formula: see text], we have proved existence of a continuous [Formula: see text] which generates an MRA with dilation factor [Formula: see text]. We have also constructed the corresponding orthonormal [Formula: see text]-wavelet.

Object detection in high resolution images based on multiscale and block processing

In the paper the algorithm for object detection in high resolution images is proposed. The approach uses multiscale image representation followed by block processing with the overlapping value. For each block the object detection with convolutional neural network was performed. Number of pyramid layers is limited by the Convolutional Neural Network layer size and input image resolution. Overlapping blocks splitting to improve the classification and detection accuracy is performed on each layer of pyramid except the highest one. Detected areas are merged into one if they have high overlapping value and the same class. Experimental results for the algorithm are presented in the paper.

Fibrillar Collagen Quantification With Curvelet Transform Based Computational Methods.

Quantification of fibrillar collagen organization has given new insight into the possible role of collagen topology in many diseases and has also identified candidate image-based bio-markers in breast cancer and pancreatic cancer. We have been developing collagen quantification tools based on the curvelet transform (CT) algorithm and have demonstrated this to be a powerful multiscale image representation method due to its unique features in collagen image denoising and fiber edge enhancement. In this paper, we present our CT-based collagen quantification software platform with a focus on new features and also giving a detailed description of curvelet-based fiber representation. These new features include C++-based code optimization for fast individual fiber tracking, Java-based synthetic fiber generator module for method validation, automatic tumor boundary generation for fiber relative quantification, parallel computing for large-scale batch mode processing, region-of-interest analysis for user-specified quantification, and pre- and post-processing modules for individual fiber visualization. We present a validation of the tracking of individual fibers and fiber orientations by using synthesized fibers generated by the synthetic fiber generator. In addition, we provide a comparison of the fiber orientation calculation on pancreatic tissue images between our tool and three other quantitative approaches. Lastly, we demonstrate the use of our software tool for the automatic tumor boundary creation and the relative alignment quantification of collagen fibers in human breast cancer pathology images, as well as the alignment quantification of in vivo mouse xenograft breast cancer images.

3D Action Recognition Using Multiscale Energy-Based Global Ternary Image

This paper presents an effective multiscale energy-based global ternary image (E-GTI) representation for action recognition from depth sequences. The unique property of our representation is that it takes the spatiotemporal discrimination and action speed variations into account, intending to solve the problems of distinguishing similar actions and identifying the actions with different speeds in one goal. The entire method is carried out in two stages. In the first stage, consecutive depth frames are used to generate global ternary image (GTI) features, which implicitly capture both inter-frame motion regions and motion directions. Specifically, each pixel in the GTI represents one of three possible states, namely, positive, negative, and neutral, which indicate the increased, decreased, and same depth values, respectively. To cope with speed variations in actions, energy-based sampling method is utilized, leading to multiscale E-GTI features, where the multiscale scheme can efficiently capture the temporal relationships among frames. In the second stage, all the E-GTI features are transformed by Radon transform (RT) as robust descriptors, which are aggregated by the bag-of-visual-words model as a compact representation. Extensive experiments on benchmark data sets show that our representation outperforms state-of-the-art approaches, since it captures discriminating spatiotemporal information of actions. Due to the merits of energy-based sampling and RT methods, our representation shows robustness to speed variations, depth noise, and partial occlusions.

Wavelet and shearlet-based image representations for visual servoing

A visual servoing scheme consists of a closed-loop control approach that uses visual information feedback to control the motion of a robotic system. Probably the most popular visual servoing method is image-based visual servoing (IBVS). This kind of method uses geometric visual features extracted from the image to design the control law. However, extracting, matching, and tracking geometric visual features over time significantly limits the versatility of visual servoing controllers in various industrial and medical applications, in particular for “low-structured” medical images, e.g. ultrasounds and optical coherence tomography modalities. To overcome the limits of conventional IBVS, one can consider novel visual servoing paradigms known as “direct” or “featureless” approaches. This paper deals with the development of a new generation of direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. In particular, we consider the use of multiscale image representations that are based on discrete wavelet and shearlet transforms. Up to now, one of the main obstacles in the investigation of multiscale image representations for visual servoing schemes was the issue of obtaining an analytical formulation of the interaction matrix that links the variation of wavelet and shearlet coefficients to the spatial velocity of the camera and the robot. In this paper, we derive four direct visual servoing controllers: two that are based on subsampled respectively non-subsampled wavelet coefficients and two that are based on the coefficients of subsampled respectively non-subsampled discrete shearlet transforms. All proposed controllers were tested in both simulation and experimental scenarios (using a six-degree-of-freedom Cartesian robot in an eye-in-hand configuration). The objective of this paper is to provide an analysis of the respective strengths and weaknesses of wavelet- and shearlet-based visual servoing controllers.

Two-dimensional gray scale image denoising via morphological operations in NSST domain & bitonic filtering

The denoising of an image is one of the most classical and basic step in image processing. The most challenging task is to design a feature preserving denoising algorithm. This article presents an efficient denoising method derived from morphological filtering in NSST domain and Bitonic filtering. In the first stage the noisy components are processed by morphological circular disc operators i.e. Top Hat/ Bottom Hat filtering in NSST domain, as Shearlet is a powerful multi-scale and multi-directional image representation tool. The resultant image is then decomposed into 8 bit planes and each bit plane is passed through bitonic filter separately. These filtered images are assembled to obtain the final denoised image. Experimental results on standard test images substantiate that the proposed method achieves reasonable and consistent denoising performance, especially in preserving fine structure information as compared with existing algorithms specifically at high noise levels.

A new image classification method using CNN transfer learning and web data augmentation

Since Convolutional Neural Network (CNN) won the image classification competition 202 (ILSVRC12), a lot of attention has been paid to deep layer CNN study. The success of CNN is attributed to its superior multi-scale high-level image representations as opposed to hand-engineering low-level features. However, estimating millions of parameters of a deep CNN requires a large number of annotated samples, which currently prevents many superior deep CNNs (such as AlexNet, VGG, ResNet) being applied to problems with limited training data. To address this problem, a novel two-phase method combining CNN transfer learning and web data augmentation is proposed. With our method, the useful feature presentation of pre-trained network can be efficiently transferred to target task, and the original dataset can be augmented with the most valuable Internet images for classification. Our method not only greatly reduces the requirement of a large training data, but also effectively expand the training dataset. Both of method features contribute to the considerable over-fitting reduction of deep CNNs on small dataset. In addition, we successfully apply Bayesian optimization to solve the tuff problem, hyper-parameter tuning, in network fine-tuning. Our solution is applied to six public small datasets. Extensive experiments show that, comparing to traditional methods, our solution can assist the popular deep CNNs to achieve better performance. Particularly, ResNet can outperform all the state-of-the-art models on six small datasets. The experiment results prove that the proposed solution will be the great tool for dealing with practice problems which are related to use deep CNNs on small dataset.

Sea Ice Drift Tracking From Sequential SAR Images Using Accelerated-KAZE Features

In this paper, we propose a feature-tracking algorithm for sea ice drift retrieval from a pair of sequential satellite synthetic aperture radar (SAR) images. The method is based on feature tracking comprising feature detection, description, and matching steps. The approach exploits the benefits of nonlinear multiscale image representations using accelerated-KAZE (A-KAZE) features, a method that detects and describes image features in an anisotropic scale space. We evaluated several state-of-the-art feature-based algorithms, including A-KAZE, Scale Invariant Feature Transform (SIFT), and a very fast feature extractor that computes binary descriptors known as Oriented FAST and Rotated BRIEF (ORB) on dual polarized Sentinel-1A C-SAR extra wide swath mode data over the Arctic. The A-KAZE approach outperforms both ORB and SIFT up to an order of magnitude in ice drift. The experimental results showed high relevance of the proposed algorithm for retrieval of ice drift at subkilometre resolution from a pair of SAR images with 100-m pixel size. From this paper, we found that feature tracking using nonlinear scale-spaces is preferable due to its high efficiency against noise with respect to image features compared with other existing feature tracking alternatives that make use of Gaussian or linear scale spaces.

Anatomical-functional image fusion by information of interest in local Laplacian filtering domain.

A novel method for performing anatomical (MRI)-functional (PET or SPECT) image fusion is presented. The method merges specific feature information from input image signals of a single or multiple medical imaging modalities into a single fused image while preserving more information and generating less distortion. The proposed method uses a local Laplacian filtering based technique realized through a novel multi-scale system architecture. Firstly, the input images are generated in a multi-scale image representation and are processed using local Laplacian filtering. Secondly, at each scale, the decomposed images are combined to produce fused approximate images using a local energy maximum scheme and produce the fused residual images using an information of interest-based scheme. Finally, a fused image is obtained using a reconstruction process that is analogous to that of conventional Laplacian pyramid transform. Experimental results computed using individual multi-scale analysis-based decomposition schemes or fusion rules clearly demonstrate the superiority of the proposed method through subjective observation as well as objective metrics. Furthermore, the proposed method can obtain better performance, compared to the state-of-the-art fusion methods.

Multi-channel and multi-scale mid-level image representation for scene classification

Convolutional neural network (CNN)-based approaches have received state-of-the-art results in scene classification. Features from the output of fully connected (FC) layers express one-dimensional semantic information but lose the detailed information of objects and the spatial information of scene categories. On the contrary, deep convolutional features have been proved to be more suitable for describing an object itself and the spatial relations among objects in an image. In addition, the feature map from each layer is max-pooled within local neighborhoods, which weakens the invariance of global consistency and is unfavorable to scenes with highly complicated variation. To cope with the above issues, an orderless multi-channel mid-level image representation on pre-trained CNN features is proposed to improve the classification performance. The mid-level image representation of two channels from the FC layer and the deep convolutional layer are integrated at multi-scale levels. A sum pooling approach is also employed to aggregate multi-scale mid-level image representation to highlight the importance of the descriptors beneficial for scene classification. Extensive experiments on SUN397 and MIT 67 indoor datasets demonstrate that the proposed method achieves promising classification performance.

Image Denoising via Multiscale Nonlinear Diffusion Models

Image denoising is a fundamental operation in image processing and holds considerable practical importance for various real-world applications. Arguably several thousands of papers are dedicated to image denoising. In the past decade, state-of-the-art denoising algorithms have been clearly dominated by nonlocal patch-based methods, which explicitly exploit patch self-similarity within the targeted image. However, in the past two years, discriminatively trained local approaches have started to outperform previous nonlocal models and have been attracting increasing attention due to the additional advantage of computational efficiency. Successful approaches include cascade of shrinkage fields (CSF) and trainable nonlinear reaction diffusion (TNRD). These two methods are built on the filter response of linear filters of small size using feed forward architectures. Due to the locality inherent in local approaches, the CSF and TNRD models become less effective when the noise level is high and consequently introduce some noise artifacts. In order to overcome this problem, in this paper we introduce a multiscale strategy. To be specific, we build on our newly developed TNRD model, adopting the multiscale pyramid image representation to devise a multiscale nonlinear diffusion process. As expected, all the parameters in the proposed multiscale diffusion model, including the filters and the influence functions across scales, are learned from training data through a loss-based approach. Numerical results on Gaussian and Poisson denoising substantiate that the exploited multiscale strategy can successfully boost the performance of the original TNRD model with a single scale. As a consequence, the resulting multiscale diffusion models can significantly suppress the typical incorrect features for those noisy images with heavy noise. It turns out that multiscale TNRD variants achieve better performance than state-of-the-art denoising methods.

A multi-scale 3D Otsu thresholding algorithm for medical image segmentation

Thresholding technique is one of the most imperative practices to accomplish image segmentation. In this paper, a novel thresholding algorithm based on 3D Otsu and multi-scale image representation is proposed for medical image segmentation. Considering the high time complexity of 3D Otsu algorithm, an acceleration variant is invented using dimension decomposition rule. In order to reduce the effects of noises and weak edges, multi-scale image representation is brought into the segmentation algorithm. The whole segmentation algorithm is designed as an iteration procedure. In each iteration, the image is segmented by the efficient 3D Otsu, and then it is filtered by a fast local Laplacian filtering to get a smoothed image which will be input into the next iteration. Finally, the segmentation results are pooled to get a final segmentation using majority voting rules. The attractive features of the algorithm are that its segmentation results are stable, it is robust to noises and it holds for both bi-level and multi-level thresholding cases. Experiments on medical MR brain images are conducted to demonstrate the effectiveness of the proposed method. The experimental results indicate that the proposed algorithm is superior to the other multilevel thresholding algorithms consistently.

Efficient tree construction for multiscale image representation and processing

With the continuous growth of sensor performances, image analysis and processing algorithms have to cope with larger and larger data volumes. Besides, the informative components of an image might not be the pixels themselves, but rather the objects they belong to. This has led to a wide range of successful multiscale techniques in image analysis and computer vision. Hierarchical representations are thus of first importance, and require efficient algorithms to be computed in order to address real-life applications. Among these hierarchical models, we focus on morphological trees (e.g., min/max-tree, tree of shape, binary partition tree, α-tree) that come with interesting properties and already led to appropriate techniques for image processing and analysis, with a growing interest from the image processing community. More precisely, we build upon two recent algorithms for efficient α-tree computation and introduce several improvements to achieve higher performance. We also discuss the impact of the data structure underlying the tree representation, and provide for the sake of illustration several applications where efficient multiscale image representation leads to fast but accurate techniques, e.g., in remote sensing image analysis or video segmentation.

Deforming Pyramid: Multiscale Image Representation Using Pixel Deformation and Filters for Non-Equispaced Signals

Deforming Pyramid: Multiscale Image Representation Using Pixel Deformation and Filters for Non-Equispaced Signals

Multi-scale features extraction from baseline structure MRI for MCI patient classification and AD early diagnosis

In this study, we investigate multi-scale features extracted from baseline structural magnetic resonance imaging (MRI) for classifying patients with mild cognitive impairment (MCI), who have either converted or not converted to Alzheimer׳s disease (AD) three years after their baseline visit. Total 549 subjects from the Alzheimer׳s disease Neuroimaging Initiative (ADNI) database are included, and there are 228 Normal controls (NC), 133 MCI patients (71 of them converted to AD within 3 years, referred as MCI converters, or MCIc) and 188 AD patients. The images are preprocessed with the standard voxel-based morphometry method with segmentation of grey matter, white matter and cerebrospinal fluid. Wavelet frame, a multi-scale image representation approach, is applied to extract features of different scales and directions from the processed grey matter image data. The features are extracted for both whole grey matter images and grey matter images of the hippocampus region. The support vector machine is adopted to construct classifiers for MCIc and MCI non-converters (MCInc). The accuracy using a leave-one-out procedure for classification of AD vs. NC and MCIc vs. MCInc is 84.13% and 76.69% respectively, both achieved by local hippocampus data. Our study shows that the proposed multi-scale method is capable of discriminating MCI converters and non-converters, and it can be potentially useful for MCI prognosis in clinical applications.

Going deeper into copy-move forgery detection: Exploring image telltales via multi-scale analysis and voting processes

This work presents a new approach toward copy-move forgery detection based on multi-scale analysis and voting processes of a digital image. Given a suspicious image, we extract interest points robust to scale and rotation finding possible correspondences among them. We cluster correspondent points into regions based on geometric constraints. Thereafter, we construct a multi-scale image representation and for each scale, we examine the generated groups using a descriptor strongly robust to rotation, scaling and partially robust to compression, which decreases the search space of duplicated regions and yields a detection map. The final decision is based on a voting process among all detection maps. We validate the method using various datasets comprising original and realistic image clonings. We compare the proposed method to 15 others from the literature and report promising results.