Multiresolution Pyramid Research Articles

Analysis of medical images super-resolution via a wavelet pyramid recursive neural network constrained by wavelet energy entropy

Recently, multi-resolution pyramid-based techniques have emerged as the prevailing research approach for image super-resolution. However, these methods typically rely on a single mode of information transmission between levels. In our approach, a wavelet pyramid recursive neural network (WPRNN) based on wavelet energy entropy (WEE) constraint is proposed. This network transmits previous-level wavelet coefficients and additional shallow coefficient features to capture local details. Besides, the parameter of low- and high-frequency wavelet coefficients within each pyramid level and across pyramid levels is shared. A multi-resolution wavelet pyramid fusion (WPF) module is devised to facilitate information transfer across network pyramid levels. Additionally, a wavelet energy entropy loss is proposed to constrain the reconstruction of wavelet coefficients from the perspective of signal energy distribution. Finally, our method achieves the competitive reconstruction performance with the minimal parameters through an extensive series of experiments conducted on publicly available datasets, which demonstrates its practical utility.

Frequency and spatial based multi-layer context network (FSCNet) for remote sensing scene classification

Remote Sensing Scene Classification (RSSC) is an important and challenging research topic due to the variety of land cover sizes and spatial combinations, as well as significant interclass similarity and intraclass variability. Currently, convolutional neural network (CNN)-based methods have been widely used in RSSC tasks with significant results. However, CNNs lack the ability to obtain long-term correlations. Transformer addressed this problem, thanks to the global receptive field of multi-head self-attention (MSA). Nevertheless, the vanilla transformer also needs further improvement to accommodate the diverse in type and scale of objects in RS scenes. In addition, the existing RSSC methods either use the last layer features, which is not conducive to process multi-scale remote sensing images, or directly fuse the multi-layer features, which will bring redundant or mutually exclusive information. To address the above issues, a novel RSSC framework, named frequency and spatial based multi-layer attention network (FSCNet) for remote sensing scene classification is proposed in this article. First, to fully extract the pyramid multi resolution features of CNN, a cross resolution injection model (CRIM) is proposed. Second, to generate better understand of the multilevel features, a frequency and spatial MLP (FS-MLP) is designed. Third, in order to aggregate contextual relations among multi-layer features, a multi-layer context align attention (MCAA) is adopted. The final classification is integration of top-layer feature and aggregated multi-layer feature. The experiment results on three well-known RS scene classification datasets (UCM, AID, and NWPU) prove the effectiveness of FSCNet and it outperforms many state-of-the-art methods.

A physics-oriented shadowgraph motion estimation algorithm for turbulent flow structures

Shadowgraph imaging is a simple, cost-effective, and non-invasive technique for visualizing strong gradients in transparent medium. This paper developed a shadowgraph motion estimation algorithm to resolve dense motion fields for turbulent flow structures. The algorithm inherits the framework of the optical flow method, which resolves the two-dimensional velocity vectors from two basic constraints using optimization techniques. In the current motion estimation scheme, a physics-oriented constraint is derived from the optical equation of shadowgraph imaging and the fluid continuity equation, while the other constraint adopts a second-order div-curl equation. The improved constraints are specifically suitable for shadowgraph imaging, which can resolve more accurate estimations. An energy function is constructed with the two constraints, and the velocity components are solved by minimizing the energy function. Several state-of-the-art optimization techniques are adopted, including multi-resolution pyramids, weighted median filter and graduated non-convex algorithms. The algorithm is applied to two test conditions covering a wide velocity range, including an acoustic excited premixed flame and a sonic jet. The comparison to Horn–Schunck optical flow indicates that the current algorithm is able to resolve more details of the turbulent flow field, and the velocity shows better continuity. The proposed algorithm has great potential for motion estimation based on shadowgraph images in high speed turbulent flow and combustion research.

Sparse and Hierarchical Transformer for Survival Analysis on Whole Slide Images.

The Transformer-based methods provide a good opportunity for modeling the global context of gigapixel whole slide image (WSI), however, there are still two main problems in applying Transformer to WSI-based survival analysis task. First, the training data for survival analysis is limited, which makes the model prone to overfitting. This problem is even worse for Transformer-based models which require large-scale data to train. Second, WSI is of extremely high resolution (up to 150,000 x 150,000 pixels) and is typically organized as a multi-resolution pyramid. Vanilla Transformer cannot model the hierarchical structure of WSI (such as patch cluster-level relationships), which makes it incapable of learning hierarchical WSI representation. To address these problems, in this paper, we propose a novel Sparse and Hierarchical Transformer (SH-Transformer) for survival analysis. Specifically, we introduce sparse self-attention to alleviate the overfitting problem, and propose a hierarchical Transformer structure to learn the hierarchical WSI representation. Experimental results based on three WSI datasets show that the proposed framework outperforms the state-of-the-art methods.

Weakly Supervised Classification for Nasopharyngeal Carcinoma with Transformer in Whole Slide Images.

Pathological examination of nasopharyngeal carcinoma (NPC) is an indispensable factor for diagnosis, guiding clinical treatment and judging prognosis. Traditional and fully supervised NPC diagnosis algorithms require manual delineation of regions of interest on the gigapixel of whole slide images (WSIs), which however is laborious and often biased. In this paper, we propose a weakly supervised framework based on Tokens-to-Token Vision Transformer (WS-T2T-ViT) for accurate NPC classification with only a slide-level label. The label of tile images is inherited from their slide-level label. Specifically, WS-T2T-ViT is composed of the multi-resolution pyramid, T2T-ViT and multi-scale attention module. The multi-resolution pyramid is designed for imitating the coarse-to-fine process of manual pathological analysis to learn features from different magnification levels. The T2T module captures the local and global features to overcome the lack of global information. The multi-scale attention module improves classification performance by weighting the contributions of different granularity levels. Extensive experiments are performed on the 802-patient NPC and CAMELYON16 dataset. WS-T2T-ViT achieves an area under the receiver operating characteristic curve (AUC) of 0.989 for NPC classification on the NPC dataset. The experiment results of CAMELYON16 dataset demonstrate the robustness and generalizability of WS-T2T-ViT in WSI-level classification.

Predictive Modeling With Multiresolution Pyramid VAE and Industrial Soft Sensor Applications.

In industrial processes, the sampling rates of process variables are discrepant because of the nature of instruments and measuring demands, which forms the challenging issue, that is, the multirate modeling in the data-driven soft sensor development. In this work, a multiresolution pyramid variational autoencoder (MR-PVAE) predictive model is proposed to solve this problem based on the deep feature extraction and feature pyramid augmentation. First, a multirate data filter is designed through a resolution searching strategy to turn the original process data into a multiresolution dataset. Then, the pyramid variational autoencoder (PVAE) is proposed to extract deep nonlinear features from the data with different resolutions. In PVAE, the augmented feature pyramid is constructed layer by layer to fuse extracted features from low resolution to the high. As a consequence, the extracted features with various resolutions are gathered to form the regression model, where the process information contained in data with discrepant sampling rates can be fully utilized. Due to the layer-by-layer enhanced features, the prediction accuracy of the soft sensing model are gradually improved. Meanwhile, an optimized training strategy is established to select the optimal feature pyramid for prediction. A numerical experiment and an industrial soft sensing case are given to validate the effectiveness and superiority of the proposed MR-PVAE model.

Self-Parameterization Based Multi-Resolution Mesh Convolution Networks

This paper addresses the challenges of designing mesh convolution neural networks for 3D mesh dense prediction. While deep learning has achieved remarkable success in image dense prediction tasks, directly applying or extending these methods to irregular graph data, such as 3D surface meshes, is nontrivial due to the non-uniform element distribution and irregular connectivity in surface meshes which make it difficult to adapt downsampling, upsampling, and convolution operations. In addition, commonly used multiresolution networks require repeated high-to-low and then low-to-high processes to boost the performance of recovering rich, high-resolution representations. To address these challenges, this paper proposes a self-parameterization-based multi-resolution convolution network that extends existing image dense prediction architectures to 3D meshes. The novelty of our approach lies in two key aspects. First, we construct a multi-resolution mesh pyramid directly from the high-resolution input data and propose area-aware mesh downsampling/upsampling operations that use sequential bijective inter-surface mappings between different mesh resolutions. The inter-surface mapping redefines the mesh, rather than reshaping it, which thus avoids introducing unnecessary errors. Second, we maintain the high-resolution representation in the multi-resolution convolution network, enabling multi-scale fusions to exchange information across parallel multi-resolution subnetworks, rather than through connections of high-to-low resolution subnetworks in series. These features differentiate our approach from most existing mesh convolution networks and enable more accurate mesh dense predictions, which is confirmed in experiments.

Multiscale Analysis for Improving Texture Classification

Information from an image occurs over multiple and distinct spatial scales. Image pyramid multiresolution representations are a useful data structure for image analysis and manipulation over a spectrum of spatial scales. This paper employs the Gaussian–Laplacian pyramid to separately treat different spatial frequency bands of a texture. First, we generate three images corresponding to three levels of the Gaussian–Laplacian pyramid for an input image to capture intrinsic details. Then, we aggregate features extracted from gray and color texture images using bioinspired texture descriptors, information-theoretic measures, gray-level co-occurrence matrix feature descriptors, and Haralick statistical feature descriptors into a single feature vector. Such an aggregation aims at producing features that characterize textures to their maximum extent, unlike employing each descriptor separately, which may lose some relevant textural information and reduce the classification performance. The experimental results on texture and histopathologic image datasets have shown the advantages of the proposed method compared to state-of-the-art approaches. Such findings emphasize the importance of multiscale image analysis and corroborate that the descriptors mentioned above are complementary.

Context-driven pyramid registration network for estimating large topology-preserved deformation

Deep learning-based deformable image registration methods have become attractive alternatives to traditional methods because of their great performance and fast run time. However, it is still challenging for these methods to estimate large topology-preserved deformation, and the contextual information that is important for large deformation is also under-mined. To address these issues, we propose a novel unsupervised context-driven pyramid registration network for estimating large topology-preserved deformation named CPRNet. Specifically, based on the multi-resolution feature pyramids, we first design multi-receptive-field guidance modules, aiming at exploiting the multi-scale spatial correlation between features of two pyramids. Then we devise multi-view context fusion modules to dynamically fuse deep contextual information containing high-level semantic information from different views of feature maps. Further, we develop a residual estimation strategy to estimate the deformation in a coarse-to-fine manner. Moreover, a deformation field regularization module is proposed to address the challenge of balancing the registration performance and topology preservation. The experiments both on liver computed tomography (CT) images and brain magnetic resonance (MR) images demonstrate that our proposed method provides effective and accurate registration on various datasets with a fast run time. Compared with existing learning-based registration methods, our proposed method exceeds the performance in most trials while maintaining desirable topology preservation capability and can potentially fit various image registration tasks.

DeriveNet for (Very) Low Resolution Image Classification.

Images captured from a distance often result in (very) low resolution (VLR/LR) region of interest, requiring automated identification. VLR/LR images (or regions of interest) often contain less information content, rendering ineffective feature extraction and classification. To this effect, this research proposes a novel DeriveNet model for VLR/LR classification, which focuses on learning effective class boundaries by utilizing the class-specific domain knowledge. DeriveNet model is jointly trained via two losses: (i) proposed Derived-Margin softmax loss and (ii) the proposed Reconstruction-Center (ReCent) loss. The Derived-Margin softmax loss focuses on learning an effective VLR classifier while explicitly modeling the inter-class variations. The ReCent loss incorporates domain information by learning a HR reconstruction space for approximating the class variations for the VLR/LR samples. It is utilized to derive inter-class margins for the Derived-Margin softmax loss. The DeriveNet model has been trained with a novel Multi-resolution Pyramid based data augmentation which enables the model to learn from varying resolutions during training. Experiments and analysis have been performed on multiple datasets for (i) VLR/LR face recognition, (ii) VLR digit classification, and (iii) VLR/LR face recognition from drone-shot videos. The DeriveNet model achieves state-of-the-art performance across different datasets, thus promoting its utility for several VLR/LR classification tasks.

Traveling Salesperson Problem with Simple Obstacles: The Role of Multidimensional Scaling and the Role of Clustering

The objective of the traveling salesperson problem (TSP) is to find a shortest tour through all nodes in a graph. Euclidean TSPs are traditionally represented with “cities” placed on a 2D plane. When straight line obstacles are added to the plane, a tour has to visit all cities while going around obstacles. The resulting problem with obstacles remains metric, but is not Euclidean because the shortest paths are no longer straight lines. We first revise a previous version of multiresolution graph pyramid by modifying the hierarchical clustering stage. Next, we describe two new experiments with human subjects. In the first experiment, the effect of the length of obstacles on the quality of tours produced by subjects was tested with three problem sizes. Long obstacles affect the tours to a greater degree than short obstacles, but long obstacles create obvious clusters and limit the ways in which the tours can be produced. In the second experiment we evaluated the degree to which Multidimensional Scaling (MDS) can compensate for the presence of obstacles. The results show that although MDS approximation can compensate to a large degree for the presence of obstacles, it cannot fully account for human performance. This fact suggests that mental representation of a TSP with obstacles is not Euclidean. Instead, it is likely to be based on hierarchical clustering in which pairwise distances represent the shortest paths around obstacles.

PDBL: Improving Histopathological Tissue Classification With Plug-and-Play Pyramidal Deep-Broad Learning.

Histopathological tissue classification is a simpler way to achieve semantic segmentation for the whole slide images, which can alleviate the requirement of pixel-level dense annotations. Existing works mostly leverage the popular CNN classification backbones in computer vision to achieve histopathological tissue classification. In this paper, we propose a super lightweight plug-and-play module, named Pyramidal Deep-Broad Learning (PDBL), for any well-trained classification backbone to improve the classification performance without a re-training burden. For each patch, we construct a multi-resolution image pyramid to obtain the pyramidal contextual information. For each level in the pyramid, we extract the multi-scale deep-broad features by our proposed Deep-Broad block (DB-block). We equip PDBL in three popular classification backbones, ShuffLeNetV2, EfficientNetb0, and ResNet50 to evaluate the effectiveness and efficiency of our proposed module on two datasets (Kather Multiclass Dataset and the LC25000 Dataset). Experimental results demonstrate the proposed PDBL can steadily improve the tissue-level classification performance for any CNN backbones, especially for the lightweight models when given a small among of training samples (less than 10%). It greatly saves the computational resources and annotation efforts. The source code is available at: https://github.com/linjiatai/PDBL.

Joint synthesis and registration network for deformable MR-CBCT image registration for neurosurgical guidance

Objective. The accuracy of navigation in minimally invasive neurosurgery is often challenged by deep brain deformations (up to 10 mm due to egress of cerebrospinal fluid during neuroendoscopic approach). We propose a deep learning-based deformable registration method to address such deformations between preoperative MR and intraoperative CBCT. Approach. The registration method uses a joint image synthesis and registration network (denoted JSR) to simultaneously synthesize MR and CBCT images to the CT domain and perform CT domain registration using a multi-resolution pyramid. JSR was first trained using a simulated dataset (simulated CBCT and simulated deformations) and then refined on real clinical images via transfer learning. The performance of the multi-resolution JSR was compared to a single-resolution architecture as well as a series of alternative registration methods (symmetric normalization (SyN), VoxelMorph, and image synthesis-based registration methods). Main results. JSR achieved median Dice coefficient (DSC) of 0.69 in deep brain structures and median target registration error (TRE) of 1.94 mm in the simulation dataset, with improvement from single-resolution architecture (median DSC = 0.68 and median TRE = 2.14 mm). Additionally, JSR achieved superior registration compared to alternative methods—e.g. SyN (median DSC = 0.54, median TRE = 2.77 mm), VoxelMorph (median DSC = 0.52, median TRE = 2.66 mm) and provided registration runtime of less than 3 s. Similarly in the clinical dataset, JSR achieved median DSC = 0.72 and median TRE = 2.05 mm. Significance. The multi-resolution JSR network resolved deep brain deformations between MR and CBCT images with performance superior to other state-of-the-art methods. The accuracy and runtime support translation of the method to further clinical studies in high-precision neurosurgery.

MultiRes-NetVLAD: Augmenting Place Recognition Training With Low-Resolution Imagery

Visual Place Recognition (VPR) is a crucial component of 6-DoF localization, visual SLAM and structure-from-motion pipelines, tasked to generate an initial list of place match hypotheses by matching global place descriptors. However, commonly-used CNN-based methods either process multiple image resolutions after training or use a single resolution and limit multi-scale feature extraction to the last convolutional layer during training. In this paper, we augment NetVLAD representation learning with low-resolution image pyramid encoding which leads to richer place representations. The resultant multi-resolution feature pyramid can be conveniently aggregated through VLAD into a single compact representation, avoiding the need for concatenation or summation of multiple patches in recent multi-scale approaches. Furthermore, we show that the underlying learnt feature tensor can be combined with existing multi-scale approaches to improve their baseline performance. Evaluation on 15 viewpoint-varying and viewpoint-consistent benchmarking datasets confirm that the proposed MultiRes-NetVLAD leads to state-of-the-art Recall@N performance for global descriptor based retrieval, compared against 11 existing techniques. Source code is publicly available at https://github.com/Ahmedest61/MultiRes-NetVLAD.

Quadruple tripatch-wise modular architecture-based real-time structure from motion

Structure from motion (SFM), a research hotspot in the intelligent transportation field (including autonomous driving, environmental perception and augmented reality (AR) for artificial intelligence terminals), can automatically recover ego-motion state estimations and 3D scene reconstructions from multiple images or video sequences. Most existing vision methods operate offline in indoor scenes, and their reconstruction accuracies greatly depend on the tracking lifetimes and accuracies of feature points. Reprojection matrix and redundant regression noise computations are exponential disasters for the calculation and reconstruction drift of large-scale scenes. This paper proposes a quadruple tripatch-wise modular architecture (QTMA) for autonomous vehicle stereo image sequences that decomposes rigid scenes into nonrigid motion-segmented pieces for reconstruction. An advanced energy function for salient image features is established by combining multiple feature types with weighted finite-element mesh. Closed quadruple annular matching and relocation are performed via multiresolution pyramid images. The proposed incremental integral-mapping calculation method and unique tree-like stacked storage containers prolong the tracking lifetimes of consecutive frames and ensure the spatiotemporal consistency and robustness of the homonymous image features in different subsequences. Experimental results verify the effectiveness of this architecture for different transportation scenes; the frame rate processing speed reaches 30 fps, the calculation accuracy regarding the path distance difference reaches 99.49%, and the estimation results regarding the maximum speeds of motion are closer to the ground truth. The translation error of the motion pose is 0.0136%, and the rotation error is 0.0035 [deg/m], which has more yaw stability than the existing state-of-the-art methods. Furthermore, in the reconstructed point cloud quality demonstration, the mean value of roughness is reduced by 40.127%, the mean value of density is improved by 27.701%, and the accuracy reaches 91.149% within a certain distance tolerance. This paper has significant theoretical research value and application potential for positioning, path tracking, and navigation in adaptive cruise control (ACC) and advanced driver assistance systems (ADAS).

Multi-resolution Pyramid Enhanced Non-local Feature Extraction for Hyperspectral Classification

Hyperspectral image (HSI) is applied to accurately distinguish ground objects in many fields, owing to its abundant structural and spectral information. To fully mine the spectral-spatial information, feature extraction methods are employed to further improve the classification performance in numerous researches. Among them, integrating contextual information into pixels is an active topic and achieved significant performance. However, it is inevitable to come with unwanted and superfluous distracting contents in the process of introducing contextual information. This paper propose a multi-resolution pyramid enhanced non-local feature extraction (MRPEN) method by exploiting non-local information from different resolutions to alleviate the effects of noise contents. The feature extraction task is implemented through by first constructing a multi-resolution feature pyramid via performing spectral-spatial reduction and resize operation on the HSI. With the feature pyramid, three kinds of feature extraction manners (i.e., short-range, long-range, and non-local) are utilized to extract specific features for each layers. In particular, non-local feature is extracted by introducing adjacent superpixels from upper layer into our feature extraction operation. After this, the extracted features of each layer are concatenated together and fed on a sparse autoencoders. Finally, a decision rule is introduced to blend the prediction results. Experiments on three public HSI datasets and a Gaofen-5 satellite dataset demonstrate that our method outperforms several state-of-the-art classification approaches.

FlowPM: Distributed TensorFlow implementation of the FastPM cosmological N-body solver

We present FlowPM, a Particle-Mesh (PM) cosmological N-body code implemented in Mesh-TensorFlow for GPU-accelerated, distributed, and differentiable simulations. We implement and validate the accuracy of a novel multi-grid scheme based on multiresolution pyramids to compute large-scale forces efficiently on distributed platforms. We explore the scaling of the simulation on large-scale supercomputers and compare it with corresponding Python based PM code, finding on an average 10x speed-up in terms of wallclock time. We also demonstrate how this novel tool can be used for efficiently solving large scale cosmological inference problems, in particular reconstruction of cosmological fields in a forward model Bayesian framework with hybrid PM and neural network forward model. We provide skeleton code for these examples and the entire code is publicly available at https://github.com/modichirag/flowpm .

Personality Trait Detection Based on ASM Localization and Deep Learning

Global competition is the competition of human resources, the social demand for high-quality talents is increasing, and the demand for all kinds of talents is increasing. Therefore, how to scientifically and efficiently complete the preliminary screening of college students’ mental health, so as to provide services for them, has become an important task. In order to solve the above problems, by combining the relevant professional knowledge of psychology, statistics, image processing, and artificial intelligence technology, a personality trait detection method based on active shape model (ASM) localization and deep learning is proposed. Firstly, the traditional ASM algorithm is improved and applied to facial feature point location, which provides training basis for further deep learning. It mainly includes three aspects of improvement: (1) 2D texture model based on Gabor wavelet and gradient features; (2) new multiresolution pyramid decomposition method; and (3) improved multiresolution pyramid search strategy. Secondly, the deep belief network model is used to train and classify the students’ four personality traits and facial features, so as to dig out the relationship between the four personality traits and facial features. The experimental results show that the localization effect of the improved ASM algorithm is obviously better than that of the traditional algorithm, and the classifier after learning and training has a good effect in analyzing the four personality traits.

Continuous Scale-Space Direct Image Alignment for Visual Odometry From RGB-D Images

In this letter, we propose a novel dense 3D image alignment algorithm that estimates the Euclidean transformation between pairs of camera poses from pixel intensities. The novelty consists in the automatic scale adaptation within each level of a multi-resolution image pyramid, using the scale-space representation of images. This is done through the continuous optimization of a scale parameter along with camera pose parameters in the same optimization framework. The proposed approach permits to significantly improve the robustness of the direct image alignment to large inter-frame motion. Various experiments on the TUM RGB-D dataset show that the proposed algorithm outperforms a fixed scale pyramid-based state-of-the-art alignment method.

Solitary Pulmonary nodule segmentation based on pyramid and improved grab cut

Background and objectiveAccurate segmentation of solitary pulmonary nodule of digital radiography image is essential for lesion appearance measurement and medical follow-up. However, the imaging characteristics of digital radiography, the inhomogeneity and fuzzy contours of nodules often lead to poor performances. This work aims to develop a segmentation framework that satisfies the requirements of accurate segmentation. MethodsIn this work, an interactive segmentation method which combined the enhanced total-variance pyramid and improved Grab cut was proposed to improve the performance of nodule segmentation. The edge-preserving multi-resolution pyramid structure did the rough segmentation on low resolution images, which provided contour nearby curves to initial the following accuracy segmentation and shortened the time of energy decreasing. With the multiscale information being incorporated to optimize the edge term and improve the appearance model, a novel Gibbs energy functional was constructed to extract the nodule in a proper scale. By introducing the multiscale processing and optimizing the energy terms, the proposed method could overcome the inhomogeneity and fuzzy contours. ResultsFor evaluation of the nodule segmentation, quantitative metrics such as precision, intersection over union, and dice similarity coefficient were introduced and compared in the experimental part. The proposed solitary pulmonary nodule segmentation method produced the results with mean values of precision 0.957, dice similarity coefficient 0.933, and intersection over union 0.891, respectively. And the corresponding standard deviation values were 0.041, 0.047, and 0.045. ConclusionsFrom the quantitative assessment and comparison in the experiments, the proposed method achieved a competitive performance in accuracy and stability, even in the cases with low contrast and fuzzy contours.