Dual Pyramid Research Articles

Multi-Scale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer

Multi-Scale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer

Deep Spatial Feature Transformation for Oriented Aerial Object Detection

Object detection in aerial images has received extensive attention in the field of computer vision. Different from natural images, the aerial objects are usually distributed in any direction. Therefore, the existing detector usually needs more parameters to encode the direction information, resulting in a large number of redundant calculations. In addition, because an ordinary convolution neural network (CNN) does not effectively model the direction change, a large amount of the rotated data is required for the aerial detector. To solve these problems, we propose a Deep Spatial Feature Transformation Network (DSFTNet), which includes a Spatial Feature Extraction Module and a Feature Selection Module. Specifically, we add the rotation convolution kernel to the detector to extract the directional feature of the rotated target to accurately predict the direction of the model. Then we build a dual pyramid to separate the features in the classification and regression tasks. Finally, the polarization function is proposed to construct the critical features that are suitable for their respective tasks, achieving feature selection and more refined detection. Experiments on public remote sensing benchmarks (e.g. DOTA, HRSC2016, and UCASAOD) have proved the effectiveness of our detector.

Multiframe Super-Resolution With Dual Pyramid Multiattention Network for Droplet Measurement

Accurate visual measurement of micrometer-scale flying droplets in inkjet printing remains a challenge due to low image resolution caused by severe image conditions. Multi-frame super resolution (MFSR) has the potential to break through the measurement bottleneck. However, most existing MFSR methods are not satisfactory in multi-frame information utilization, especially for fast-motion scenes, and they often suffer from detail loss. In this study, focusing on multi-frame information utilization and deep feature extraction, we propose a dual pyramid multi-attention network (DPMAN). First, a dual pyramid deformable alignment (DPDA) module is proposed to deal with diverse motion, which extracts explicit offsets to enhance deformable alignment and perform coarse-to-fine alignment. Then, a gated attention fusion (GAF) module is devised to adaptively aggregate the aligned features to emphasize favorable features. Finally, a residual self-attention reconstruction (RSAR) module based on the multi-stage aggregation self-attention architecture is proposed to extract finer deep features for detail restoration. Experimental results on three benchmark datasets demonstrate that DPMAN achieves state-of-the-art performance. DPMAN is applied to droplet image reconstruction and improves the measurement accuracy from 3.34% to 2.52%.

Hierarchical spatio-spectral fusion for hyperspectral image super resolution via sparse representation and pre-trained deep model

Fusing a hyperspectral image (HSI) with a high resolution multispectral image (MSI) has been a highly attractive and effective approach for improving the spatial resolution of HSIs. However, most existing spatio-spectral fusion methods directly accomplish such a fusion process on the desired image scale. This limits the accuracy of the resulting model, especially when large magnification factors are involved. In this paper, a new dual pyramid model is proposed to implement hyperspectral image super resolution, based on a novel hierarchical spatial and spectral fusion method, in an effort to estimate the high resolution image progressively. In particular, an input low resolution HSI is upscaled progressively using a pre-trained deep Laplacian pyramid network, while the corresponding high resolution MSI is down sampled with multiple pyramid layers, forming a dual pyramid model. At each pyramid layer, the HSI and the MSI within the current layer are fused via sparse representation. Systematic qualitative and quantitative evaluations against benchmark datasets demonstrate that this new approach outperforms a number of spatio-spectral fusion based super resolution techniques, achieving outstanding performance over large scale factors.

Influence of Β-Hydroxy-Β-Methyl Butyrate Supplementation on Strength, Muscle and Liver-Damage Indices Induced by Dual Pyramid Resistance Training in Beginner Bodybuilders

Influence of Β-Hydroxy-Β-Methyl Butyrate Supplementation on Strength, Muscle and Liver-Damage Indices Induced by Dual Pyramid Resistance Training in Beginner Bodybuilders

Spatio-Temporal Causal Transformer for Multi-Grained Surgical Phase Recognition.

Automatic surgical phase recognition plays a key role in surgical workflow analysis and overall optimization in clinical work. In the complicated surgical procedures, similar inter-class appearance and drastic variability in phase duration make this still a challenging task. In this paper, a spatio-temporal transformer is proposed for online surgical phase recognition with different granularity. To extract rich spatial information, a spatial transformer is used to model global spatial dependencies of each time index. To overcome the variability in phase duration, a temporal transformer captures the multi-scale temporal context of different time indexes with a dual pyramid pattern. Our method is thoroughly validated on the public Cholec80 dataset with 7 coarse-grained phases and the CATARACTS2020 dataset with 19 fine-grained phases, outperforming state-of-the-art approaches with 91.4% and 84.2% accuracy, taking only 24.5M parameters.

SA-DPNet: Structure-aware dual pyramid network for salient object detection

Salient object detection aims at highlighting the most visually distinctive objects in the scene. Previous deep learning based works mainly focus on designing different integration strategies of multi-level features to improve the quality of prediction. However, due to the negligence of spatial structure coherence in predicted saliency maps, they fail to produce satisfactory results in complex scenarios. In this work, we present a structure-aware dual pyramid network (SA-DPNet) for salient object detection. By explicitly formulating spatial location information and spatial covariance features into the self-attention mechanism, a structure-aware spatial non-local block is proposed in SA-DPNet to learn the spatial-sensitive global context. With the proposed edge loss and adversarial loss, the edge structure context and patch-based global structure context are introduced to refine the structural coherence of the predicted results. Comprehensive experimental results on six RGB saliency benchmark datasets and three RGB-D saliency benchmark datasets demonstrate the superiority of proposed SA-DPNet over other state-of-the-art methods, both quantitatively and visually.

Ground-Based Remote Sensing Cloud Detection Using Dual Pyramid Network and Encoder–Decoder Constraint

Many methods for ground-based remote sensing cloud detection learn representation features using the encoder–decoder structure. However, they only consider the information from single scale, which leads to incomplete feature extraction. In this article, we propose a novel deep network named dual pyramid network (DPNet) for ground-based remote sensing cloud detection, which possesses an encoder–decoder structure with dual pyramid pooling module (DPPM). Specifically, we process the feature maps of different scales in the encoder through dual pyramid pooling. Then, we fuse the outputs of the dual pyramid pooling in the same pyramid level using the attention fusion. Furthermore, we propose the encoder–decoder constraint (EDC) to relieve information loss in the process of encoding and decoding. It constrains the values and the gradients of probability maps from the encoder and the decoder to be consistent. Since the number of cloud images in the publicly available databases for ground-based remote sensing cloud detection is limited, we release the TJNU Large-scale Cloud Detection Database (TLCDD) that is the largest database in this field. We conduct a series of experiments on TLCDD, and the experimental results verify the effectiveness of the proposed method.

Learning-based dose prediction for pancreatic stereotactic body radiation therapy using dual pyramid adversarial network

Treatment planning for pancreatic cancer stereotactic body radiation therapy (SBRT) is very challenging owing to vast spatial variations and close proximity of many organs-at-risk. Recently, deep learning (DL) based methods have been applied in dose prediction tasks of various treatment sites with the aim of relieving planning challenges. However, its effectiveness on pancreatic cancer SBRT is yet to be fully explored due to limited investigations in the literature. This study aims to further current knowledge in DL-based dose prediction tasks by implementing and demonstrating the feasibility of a new dual pyramid networks (DPNs) integrated DL model for predicting dose distributions of pancreatic SBRT. The proposed framework is composed of four parts: CT-only feature pyramid network (FPN), contour-only FPN, late fusion network and an adversarial network. During each phase of the network, combination of mean absolute error, gradient difference error, histogram matching, and adversarial loss is used for supervision. The performance of proposed model was demonstrated for pancreatic cancer SBRT plans with doses prescribed between 33 and 50 Gy across as many as three planning target volumes (PTVs) in five fractions. Five-fold cross validation was performed on 30 patients, and another 20 patients were used as holdout tests of trained model. Predicted plans were compared with clinically approved plans through dose volume parameters and two-paired t-test. For the same sets, our results were compared with three different DL architectures: 3D U-Net, 3D U-Net with adversarial learning, and DPN without adversarial learning. The proposed framework was able to predict 87% and 91% of clinically relevant dose parameters for cross validation sets and holdout sets, respectively, without any significant differences (P > 0.05). Dose distribution predicted by our framework was also able to predict the intentional hotspots as feature characteristics of SBRT plans. Our method achieved higher correlation coefficients with the ground truth in 22/26, 24/26 and 20/26 dose volume parameters compared to the network without adversarial learning, 3D U-Net, and 3D U-Net with adversarial learning, respectively. Overall, the proposed model was able to predict doses to cases with both single and multiple PTVs. In conclusion, the DPN integrated DL model was successfully implemented, and demonstrated good dose prediction accuracy and dosimetric characteristics for pancreatic cancer SBRT.

Head and neck multi-organ segmentation on dual-energy CT using dual pyramid convolutional neural networks

Organ delineation is crucial to diagnosis and therapy, while it is also labor-intensive and observer-dependent. Dual energy CT (DECT) provides additional image contrast than conventional single energy CT (SECT), which may facilitate automatic organ segmentation. This work aims to develop an automatic multi-organ segmentation approach using deep learning for head-and-neck region on DECT. We proposed a mask scoring regional convolutional neural network (R-CNN) where comprehensive features are firstly learnt from two independent pyramid networks and are then combined via deep attention strategy to highlight the informative ones extracted from both two channels of low and high energy CT. To perform multi-organ segmentation and avoid misclassification, a mask scoring subnetwork was integrated into the Mask R-CNN framework to build the correlation between the class of potential detected organ’s region-of-interest (ROI) and the shape of that organ’s segmentation within that ROI. We evaluated our model on DECT images from 127 head-and-neck cancer patients (66 training, 61 testing) with manual contours of 19 organs as training target and ground truth. For large- and mid-sized organs such as brain and parotid, the proposed method successfully achieved average Dice similarity coefficient (DSC) larger than 0.8. For small-sized organs with very low contrast such as chiasm, cochlea, lens and optic nerves, the DSCs ranged between around 0.5 and 0.8. With the proposed method, using DECT images outperforms using SECT in almost all 19 organs with statistical significance in DSC (p<0.05). Meanwhile, by using the DECT, the proposed method is also significantly superior to a recently developed FCN-based method in most of organs in terms of DSC and the 95th percentile Hausdorff distance. Quantitative results demonstrated the feasibility of the proposed method, the superiority of using DECT to SECT, and the advantage of the proposed R-CNN over FCN on the head-and-neck patient study. The proposed method has the potential to facilitate the current head-and-neck cancer radiation therapy workflow in treatment planning.

Male pelvic CT multi-organ segmentation using synthetic MRI-aided dual pyramid networks

The delineation of the prostate and organs-at-risk (OARs) is fundamental to prostate radiation treatment planning, but is currently labor-intensive and observer-dependent. We aimed to develop an automated computed tomography (CT)-based multi-organ (bladder, prostate, rectum, left and right femoral heads (RFHs)) segmentation method for prostate radiation therapy treatment planning. The proposed method uses synthetic MRIs (sMRIs) to offer superior soft-tissue information for male pelvic CT images. Cycle-consistent adversarial networks (CycleGAN) were used to generate CT-based sMRIs. Dual pyramid networks (DPNs) extracted features from both CTs and sMRIs. A deep attention strategy was integrated into the DPNs to select the most relevant features from both CTs and sMRIs to identify organ boundaries. The CT-based sMRI generated from our previously trained CycleGAN and its corresponding CT images were inputted to the proposed DPNs to provide complementary information for pelvic multi-organ segmentation. The proposed method was trained and evaluated using datasets from 140 patients with prostate cancer, and were then compared against state-of-art methods. The Dice similarity coefficients and mean surface distances between our results and ground truth were 0.95 ± 0.05, 1.16 ± 0.70 mm; 0.88 ± 0.08, 1.64 ± 1.26 mm; 0.90 ± 0.04, 1.27 ± 0.48 mm; 0.95 ± 0.04, 1.08 ± 1.29 mm; and 0.95 ± 0.04, 1.11 ± 1.49 mm for bladder, prostate, rectum, left and RFHs, respectively. Mean center of mass distances was within 3 mm for all organs. Our results performed significantly better than those of competing methods in most evaluation metrics. We demonstrated the feasibility of sMRI-aided DPNs for multi-organ segmentation on pelvic CT images, and its superiority over other networks. The proposed method could be used in routine prostate cancer radiotherapy treatment planning to rapidly segment the prostate and standard OARs.

Head-and-neck organs-at-risk auto-delineation using dual pyramid networks for CBCT-guided adaptive radiotherapy

Organ-at-risk (OAR) delineation is a key step for cone-beam CT (CBCT) based adaptive radiotherapy planning that can be a time-consuming, labor-intensive, and subject-to-variability process. We aim to develop a fully automated approach aided by synthetic MRI for rapid and accurate CBCT multi-organ contouring in head-and-neck (HN) cancer patients. MRI has superb soft-tissue contrasts, while CBCT offers bony-structure contrasts. Using the complementary information provided by MRI and CBCT is expected to enable accurate multi-organ segmentation in HN cancer patients. In our proposed method, MR images are firstly synthesized using a pre-trained cycle-consistent generative adversarial network given CBCT. The features of CBCT and synthetic MRI (sMRI) are then extracted using dual pyramid networks for final delineation of organs. CBCT images and their corresponding manual contours were used as pairs to train and test the proposed model. Quantitative metrics including Dice similarity coefficient (DSC), Hausdorff distance 95% (HD95), mean surface distance, and residual mean square distance (RMS) were used to evaluate the proposed method. The proposed method was evaluated on a cohort of 65 HN cancer patients. CBCT images were collected from those patients who received proton therapy. Overall, DSC values of 0.87 ± 0.03, 0.79 ± 0.10/0.79 ± 0.11, 0.89 ± 0.08/0.89 ± 0.07, 0.90 ± 0.08, 0.75 ± 0.06/0.77 ± 0.06, 0.86 ± 0.13, 0.66 ± 0.14, 0.78 ± 0.05/0.77 ± 0.04, 0.96 ± 0.04, 0.89 ± 0.04/0.89 ± 0.04, 0.83 ± 0.02, and 0.84 ± 0.07 for commonly used OARs for treatment planning including brain stem, left/right cochlea, left/right eye, larynx, left/right lens, mandible, optic chiasm, left/right optic nerve, oral cavity, left/right parotid, pharynx, and spinal cord, respectively, were achieved. This study provides a rapid and accurate OAR auto-delineation approach, which can be used for adaptive radiation therapy.

Iterative Knowledge Distillation for Automatic Check-Out

Automatic Check-Out (ACO) provides an object detection based mechanism for retailers to process the purchases of customers automatically. However, it suffers a lot from the domain shift problem because of different data distribution between the single item in training exemplar images and mixed items in testing checkout images. In this paper, we propose a new iterative knowledge distillation method to solve the domain adaptation problem for this task. First, we develop a new augmentation data strategy to generate synthesized checkout images. It can extract segmented items from the training images by the coarse-to-fine strategy and filter items with unrealistic poses by pose pruning. Second, we propose a dual pyramid scale network (DPSNet) to exploit the multi-scale feature representation in joint detection and counting views. Third, the iterative knowledge distillation training strategy is developed to make full use of both image-level and instance-level samples to narrow the semantic gap between source domain and target domain. Extensive experiments on the large-scale Retail Product Checkout (RPC) dataset show the proposed DPSNet can achieve state-of-the-art performance compared with existing methods. The source codes can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://isrc.iscas.ac.cn/gitlab/research/dpsnet</uri> .

Head and neck multi-organ auto-segmentation on CT images aided by synthetic MRI.

Because the manual contouring process is labor-intensive and time-consuming, segmentation of organs-at-risk (OARs) is a weak link in radiotherapy treatment planning process. Our goal was to develop a synthetic MR (sMR)-aided dual pyramid network (DPN) for rapid and accurate head and neck multi-organ segmentation in order to expedite the treatment planning process. Forty-five patients' CT, MR, and manual contours pairs were included as our training dataset. Nineteen OARs were target organs to be segmented. The proposed sMR-aided DPN method featured a deep attention strategy to effectively segment multiple organs. The performance of sMR-aided DPN method was evaluated using five metrics, including Dice similarity coefficient (DSC), Hausdorff distance 95% (HD95), mean surface distance (MSD), residual mean square distance (RMSD), and volume difference. Our method was further validated using the 2015 head and neck challenge data. The contours generated by the proposed method closely resemble the ground truth manual contours, as evidenced by encouraging quantitative results in terms of DSC using the 2015 head and neck challenge data. Mean DSC values of 0.91±0.02, 0.73±0.11, 0.96±0.01, 0.78±0.09/0.78±0.11, 0.88±0.04/0.88±0.06 and 0.86±0.08/0.85±0.1 were achieved for brain stem, chiasm, mandible, left/right optic nerve, left/right parotid, and left/right submandibular, respectively. We demonstrated the feasibility of sMR-aided DPN for head and neck multi-organ delineation on CT images. Our method has shown superiority over the other methods on the 2015 head and neck challenge data results. The proposed method could significantly expedite the treatment planning process by rapidly segmenting multiple OARs.

Parsing human image by fusing semantic and spatial features: A deep learning approach

How to parse the human image to obtain the text label corresponding to the human body is a critical task for human-computer interaction. Although previous methods have significantly improved the parsing performance, the problem of parsing confusion and tiny target missing remains unresolved, which leads to errors and incomplete inference accordingly. Targeting at these drawbacks, we fuse semantic and spatial features to mine the human body information based on the Dual Pyramid Unit convolutional neural network, named as DPUNet. DPUNet is composed of Context Pyramid Unit (CPU) and Spatial Pyramid Unit (SPU). Firstly, we design the CPU to aggregate the local to global semantic information, which exports the semantic feature for eliminating the semantic confusion. To capture the tiny targets for preventing the details from missing, the SPU is proposed to incorporate the multi-scale spatial information and output the spatial feature. Finally, the features of two complementary units are fused for accurate and complete human parsing results. Our approach achieves more excellent performance than the state-of-the-art methods on single human and multiple human parsing datasets. Meanwhile, the proposed framework is efficient with a fast speed of 41.2fps.

Dual pyramid network for salient object detection

Saliency detection is a fundamental yet challenging task in computer vision, aiming at identifying the most visually distinctive objects in an image. Despite promising results from recent deep learning based methods, existing works are still not sufficient to incorporate momentous global context to detect salient objects. In this work, we present a dual pyramid network (DPNet) for salient object detection by formulating self-attention mechanism into the sub-regions based contexts. Our key idea is to aggregate different sub-regions based contexts into the enhanced global context without sacrificing the resolution of deep features. To achieve this, we design the spatial pyramid module and channel-wise pyramid module. The proposed modules can enhance the global context by incorporating self-attention mechanism into the sub-regions based contexts in the spatial and channel-wise dimension, respectively. Compared with 23 state-of-the-art methods on six RGB saliency benchmark datasets, experimental results show that our method performs favorably over all the others, both quantitatively and visually. The superior performance of proposed DPNet than others on three RGB-D saliency benchmark datasets further demonstrates the powerful generalization ability of our network.

Multi-Component Fusion Network for Small Object Detection in Remote Sensing Images

Small object detection is a major challenge in the field of object detection. With the development of deep learning, many methods based on deep convolutional neural networks (DCNNs) have greatly improved the speed of detection while ensuring accuracy. However, due to the contradiction between the spatial details and semantic information of DCNNs, previous deep learning methods often meet problems when detecting small objects. The challenge can be more serious in complex scenes involving similar background objects and/or occlusion, such as in remote sensing imagery. In this paper, we propose an end-to-end DCNN called the multi-component fusion network (MCFN) to improve the accuracy of small object detection in such cases. First, we propose a dual pyramid fusion network, which densely concatenates spatial information and semantic information to extract small object features via encoding and decoding operations. Then we use a relative region proposal network to adequately extract the features of small objects samples and parts of objects. Finally, to achieve robustness against background disturbance, we add contextual information to the proposal regions before final detection. Experimental evaluations demonstrate that the proposed method significantly improves the accuracy of object detection in remote sensing images compared with other state-of-the-art methods, especially in complex scenes with the conditions of occlusion.

Identifying the Conditions for Rural Sustainability through Place-Based Culture: Applying the CIPM and CDPM Models into Meibei Ancient Village

Transitional rural China faces more serious challenges in its sustainable development. How to regain the vital momentum of those historically and culturally preeminent villages, among over 680,000 administrative villages in total, has become the pressing agenda for all the stakeholders, due to the fact that these villages have huge potential to be the leverage for successful rural transition and new urbanization in China. This paper therefore tries to diagnose and identify the current situation of those villages from a cultural perspective by taking the Meibei ancient village as the case. By applying the proposed Cultural Inverted Pyramid Model (CIPM) and Cultural Dual Pyramid Model (CDPM) with seven layers, i.e., root/vision, value, symbol, hero, ritual, lifestyle, and governance &amp; management, Meibei’s development mechanism has been systematically explored from a cultural perspective through the comparison between its past prosperity and present challenges. It is found that the great merit of Meibei’s past prosperity lied in the organic integration of cultural elements in all the layers through the five development dimensions, i.e., economic, social, institutional, environmental and cultural dimensions. The empirical study proves that CIPM is a useful tool for diagnosing and identifying the current situation of the village, while CDPM is an effective instrument for planning and designing a culture-embedded and improved place for the future. Unless Meibei can recreate a new cultural ecosystem with resilience fitting to its existed heritage with cultural excellence and tourism promotion, the village cannot catch up with its past prosperity. Finally, this paper calls for more in-depth culture-oriented research to improve the CIPM and CDPM paradigm to allow for the realization of rural sustainability, particularly from the perspectives of policy options and academic concerns.

Comparison of the effect of two selected resistance training patterns on some physical and physiological factors of elite freestyle wrestler young boys

Background: Wrestling is a sport heavy and severe activity due to certain physical and physiological needs such as anaerobic, power, strength, speed, lactate tolerance, and anaerobic stamina because wrestling matches are carried out at different weights. A more successful wrestling is a more effective athletic performance for repeated exercises and endurance fatigue. Aims and Objectives: The main objective of this research was to compare the effect of two resistance training methods (dual pyramid and diagonal pyramid) on some physical and physiological factors of elite freestyle wrestler’s young boy in Hamadan province. Materials and Methods: Subjects were divided into three groups of 10 (Experimental I Dual Pyramid Practices, Experimental II Diagonal Pyramid) and control group. The three groups trained for every 12 weeks and each week had 3 sessions. In terms of the method of research, three groups participated in a 12 week exercise program with three sessions per week. Body combination, fat index, muscular density, strength, muscular endurance and anaerobic capabilities of wrestlers were measured both prior and post to the intervention program using the Kolmogorov–Smirnov, Multivariate Analysis of Covariance (MANCOVA) in addition to Sidak Post-Hoc Test. Results: The results of the statistical test showed that there is a significant difference between the two groups in power of muscle strength, muscular endurance, power, and fat percentage (P ≤ 0.05). Conclusion: The results of this study suggest that two different exercise programs produce similar results in strength, endurance, and muscle mass, but it may have more effectiveness to improve the ability of diagonal training programs and to improve the body composition and reduce the percentage of fat and pyramid training programs.

A dual pyramid watermarking for JPEG-2000

In this paper, we propose a dual pyramid watermarking method for JPEG-2000. The robust pyramid watermark is embedded into the host image by changing the wavelet coefficients according to the characteristics of wavelet transformation, regions of interest, and Embedded Block Coding with Optimised Truncation (EBCOT) to protect copyrights. The fragile pyramid watermark records the characteristics of the coefficients in the high-high band to detect slight changes and authenticate the owner of the images. Experimental results show that the proposed technique successfully survives JPEG-2000 compression, image cropping, pepper noise, scaling and rotation and can detect modification of the host image.