High-level Map Research Articles

DA-FPN: Deformable Convolution and Feature Alignment for Object Detection

This study sought to address the problem of the insufficient extraction of shallow object information and boundary information when using traditional FPN structures in current object detection algorithms, which degrades object detection accuracy. In this paper, a new FPN structure model, DA-FPN, is proposed. DA-FPN replaces the 1 × 1 convolution used in the conventional FPN structure for lateral connection with a 3 × 3 deformable convolution and adds a feature alignment module after the 2x downsampling operation used for lateral connection. This design allows the detection framework to extract more accurate information about the boundary of the object, particularly the boundary information of small objects. A bottom-up module was also added to incorporate the shallow information of the object more accurately into the high-level feature map, and a feature alignment module was added to the bottom-up module, thereby improving object detection accuracy. The experimental results show that DA-FPN can improve the accuracy of the single-stage object detection algorithms FoveaBox and GFL by 1.7% and 2.4%, respectively, on the MS-COCO dataset. This model was also found to improve the two-stage object detection algorithm SABL by 2.4% and offer higher small object detection accuracy and better robustness.

Attention-based dual-branch deep network for sparse-view computed tomography image reconstruction.

The widespread application of X-ray computed tomography (CT) imaging in medical screening makes radiation safety a major concern for public health. Sparse-view CT is a promising solution to reduce the radiation dose. However, the reconstructed CT images obtained using sparse-view CT may suffer severe streaking artifacts and structural information loss. In this study, a novel attention-based dual-branch network (ADB-Net) is proposed to solve the ill-posed problem of sparse-view CT image reconstruction. In this network, downsampled sinogram input is processed through 2 parallel branches (CT branch and signogram branch) of the ADB-Net to independently extract the distinct, high-level feature maps. These feature maps are fused in a specified attention module from 3 perspectives (channel, plane, and spatial) to allow complementary optimizations that can mitigate the streaking artifacts and the structure loss in sparse-view CT imaging. Numerical simulations, an anthropomorphic thorax phantom, and in vivo preclinical experiments were conducted to verify the sparse-view CT imaging performance of the ADB-Net. The proposed network achieved a root-mean-square error (RMSE) of 20.6160, a structural similarity (SSIM) of 0.9257, and a peak signal-to-noise ratio (PSNR) of 38.8246 on numerical data. The visualization results demonstrate that this newly developed network can consistently remove the streaking artifacts while maintaining the fine structures. The proposed attention-based dual-branch deep network, ADB-Net, provides a promising alternative to reconstruct high-quality sparse-view CT images for low-dose CT imaging.

Application of Feature Pyramid Network and Feature Fusion Single Shot Multibox Detector for Real-Time Prostate Capsule Detection

In the process of feature propagation, the low-level convolution layers of the forward feature propagation network lack semantic information, and information loss occurs when fine-grained information is transferred to higher-level convolution; therefore, multi-stage feature fusion networks are needed to solve the interaction between low-level convolution layers and high-level convolution layers. Based on a two-way feature feedback network and feature fusion mechanism, we created a new object detection network called Feature Pyramid Network (FPN)-based Feature Fusion Single Shot Multibox Detector (FFSSD). A bottom-up and top-down architecture with lateral connections enhances the detector’s ability to extract features, then high-level multi-scale semantic feature maps are utilized to generate a feature pyramid network. The results show that the proposed network the mAP for prostate capsule image detection reaches 83.58%, providing real-time detection ability. The context interaction mechanism can transfer high-level semantic information to low-level convolution, and the resulting convolution after low-level and high-level fusion contains richer location and semantic information.

Cancer detection for small-size and ambiguous tumors based on semantic FPN and transformer.

Early detection of tumors has great significance for formative detection and determination of treatment plans. However, cancer detection remains a challenging task due to the interference of diseased tissue, the diversity of mass scales, and the ambiguity of tumor boundaries. It is difficult to extract the features of small-sized tumors and tumor boundaries, so semantic information of high-level feature maps is needed to enrich the regional features and local attention features of tumors. To solve the problems of small tumor objects and lack of contextual features, this paper proposes a novel Semantic Pyramid Network with a Transformer Self-attention, named SPN-TS, for tumor detection. Specifically, the paper first designs a new Feature Pyramid Network in the feature extraction stage. It changes the traditional cross-layer connection scheme and focuses on enriching the features of small-sized tumor regions. Then, we introduce the transformer attention mechanism into the framework to learn the local feature of tumor boundaries. Extensive experimental evaluations were performed on the publicly available CBIS-DDSM dataset, which is a Curated Breast Imaging Subset of the Digital Database for Screening Mammography. The proposed method achieved better performance in these models, achieving 93.26% sensitivity, 95.26% specificity, 96.78% accuracy, and 87.27% Matthews Correlation Coefficient (MCC) value, respectively. The method can achieve the best detection performance by effectively solving the difficulties of small objects and boundaries ambiguity. The algorithm can further promote the detection of other diseases in the future, and also provide algorithmic references for the general object detection field.

Feature-guided attention network for medical image segmentation.

U-Net and its variations have achieved remarkable performances in medical image segmentation. However, they have two limitations. First, the shallow layer feature of the encoder always contains background noise. Second, semantic gaps exist between the features of the encoder and the decoder. Skip-connections directly connect the encoder to the decoder, which will lead to the fusion of semantically dissimilar featuremaps. To overcome these two limitations, this paper proposes a novel medical image segmentation algorithm, called feature-guided attention network, which consists of U-Net, the cross-level attention filtering module (CAFM), and the attention-guided upsampling module (AUM). In the proposed method, the AUM and the CAFM were introduced into the U-Net, where the AUM learns to filter the background noise in the low-level feature map of the encoder and the CAFM tries to eliminate the semantic gap between the encoder and the decoder. Specifically, the AUM adopts a top-down pathway to use the high-level feature map so as to filter the background noise in the low-level feature map of the encoder. The AUM uses the encoder features to guide the upsampling of the corresponding decoder features, thus eliminating the semantic gap between them. Four medical image segmentation tasks, including coronary atherosclerotic plaque segmentation (Dataset A), retinal vessel segmentation (Dataset B), skin lesion segmentation (Dataset C), and multiclass retinal edema lesions segmentation (Dataset D), were used to validate the proposedmethod. For Dataset A, the proposed method achieved higher Intersection over Union (IoU) ( ), dice ( ), accuracy ( ), and sensitivity ( ) than the previous best method: CA-Net. For Dataset B, the proposed method achieved higher sensitivity (83.50%) and accuracy (97.55%) than the previous best method: SCS-Net. For Dataset C, the proposed method had highest IoU ( ) and dice ( ) than those of all compared previous methods. For Dataset D, the proposed method had highest dice (average: 81.53%; retina edema area [REA]: 83.78%; pigment epithelial detachment [PED] 77.13%), sensitivity (REA: 89.01%; SRF: 85.50%), specificity (REA: 99.35%; PED: 100.00), and accuracy (98.73%) among all compared previous networks. In addition, the number of parameters of the proposed method was 2.43 M, which is less than CA-Net (3.21 M) and CPF-Net (3.07 M). The proposed method demonstrated state-of-the-art performance, outperforming other top-notch medical image segmentation algorithms. The CAFM filtered the background noise in the low-level feature map of the encoder, while the AUM eliminated the semantic gap between the encoder and the decoder. Furthermore, the proposed method was of high computationalefficiency.

Local Feature Search Network for Building and Water Segmentation of Remote Sensing Image

Extracting buildings and water bodies from high-resolution remote sensing images is of great significance for urban development planning. However, when studying buildings and water bodies through high-resolution remote sensing images, water bodies are very easy to be confused with the spectra of dark objects such as building shadows, asphalt roads and dense vegetation. The existing semantic segmentation methods do not pay enough attention to the local feature information between horizontal direction and position, which leads to the problem of misjudgment of buildings and loss of local information of water area. In order to improve this problem, this paper proposes a local feature search network (DFSNet) application in remote sensing image building and water segmentation. By paying more attention to the local feature information between horizontal direction and position, we can reduce the problems of misjudgment of buildings and loss of local information of water bodies. The discarding attention module (DAM) introduced in this paper reads sensitive information through direction and location, and proposes the slice pooling module (SPM) to obtain a large receptive field in the pixel by pixel prediction task through parallel pooling operation, so as to reduce the misjudgment of large areas of buildings and the edge blurring in the process of water body segmentation. The fusion attention up sampling module (FAUM) guides the backbone network to obtain local information between horizontal directions and positions in spatial dimensions, provide better pixel level attention for high-level feature maps, and obtain more detailed segmentation output. The experimental results of our method on building and water data sets show that compared with the existing classical semantic segmentation model, the proposed method achieves 2.89% improvement on the indicator MIoU, and the final MIoU reaches 83.73%.

Feature Fusion Pyramid Network for End-to-end Scene Text Detection

How to properly involve text characteristics like multi-scale, arbitrary direction, length aspect ratio, into detection network design has become a hot topic in computer vision. Feature Pyramid Network (FPN) is a typical method to achieve robust text detection, where its low-level and high-level feature map retains spatial structure and global semantic information, respectively. However, its strict hierarchical structure fails to fuse low-level and high-level information to improve distinguish ability of feature map. To address this problem, we propose a novel feature fusion pyramid network for end-to-end scene text detection by fusing multi-modal information. By diving pyramid structure into high-level and low-level layers, channel and spatial attention modules are adopted to enhance high-level and low-level feature representation by encoding channel and spatial -wise context information, respectively. In order to reduce information loss by layer transmission, a special residual network is designed to achieve short-cut between high-level and low-level features, so as to realize multi-modal feature fusion. Experiments show the precision and recall of the propose method on ICDAR2015, ICDAR2017-MLT and MSRA-TD500 datasets reach 88.7%/82.1%, 77.0%/60.3% and 85.3%/74.8%, respectively.

MiniGAN: Toward Informative and Uninformative Image Transferring

This article proposes a generative adversarial networks (MiniGAN) to tackle both informative and uninformative image transferring. The generator of MiniGAN is based on the structure of StyleGANv2, in which the encoder and style transform block are proposed to extract the high-level feature maps of the source image and capture the latent representation of the target image, respectively. This information guides the generator for the final image generation. The proposed MiniGAN outperforms other models in style transferring while preserving the color information on the informative images. To test the performance of MiniGAN on the uninformative images, a new data set consisting of 10,000 fashion hand drawings is proposed. Extensive experiments and detailed analysis are presented to demonstrate the performance of MiniGAN.

Micro-Expression Spotting Based on a Short-Duration Prior and Multi-Stage Feature Extraction

When micro-expressions are mixed with normal or macro-expressions, it becomes increasingly challenging to spot them in long videos. Aiming at the specific time prior of micro-expressions (MEs), an ME spotting network called AEM-Net (adaptive enhanced ME detection network) is proposed. This paper is an extension of the conference paper presented at the Chinese Conference on Biometric Recognition (CCBR). The network improves spotting performance in the following five aspects. Firstly, a multi-stage channel feature extraction module is constructed to extract the features at different depths. Then, an attention spatial-temporal module is leveraged to obtain salient and discriminative micro-expression segments while suppressing the generation of excessively long or short suggestions. Thirdly, a ME-NMS (non-maximum suppression) network is developed to reduce redundancy and decision errors. Fourthly, a multi-scale feature fusion module is introduced to fuse up-sampling features of high-level maps and fine-grained information, which obtains meaningful information on feature distribution and contributes to a good representation of MEs. Finally, two spotting mechanisms named anchor-based and anchor free were integrated to get final spotting. Extensive experiments were conducted on prevalent CAS(ME)2 and the SAMM-Long ME databases to evaluate the spotting performance. The results show that the AEM-Net achieves competitive performance, outperforming other state-of-the-art methods.

Twist-Net: A multi-modality transfer learning network with the hybrid bilateral encoder for hypopharyngeal cancer segmentation

Hypopharyngeal cancer (HPC) is a rare disease. Therefore, it is a challenge to automatically segment HPC tumors and metastatic lymph nodes (HPC risk areas) from medical images with the small-scale dataset. Combining low-level details and high-level semantics from feature maps in different scales can improve the accuracy of segmentation. Herein, we propose a Multi-Modality Transfer Learning Network with Hybrid Bilateral Encoder (Twist-Net) for Hypopharyngeal Cancer Segmentation. Specifically, we propose a Bilateral Transition (BT) block and a Bilateral Gather (BG) block to twist (fuse) high-level semantic feature maps and low-level detailed feature maps. We design a block with multi-receptive field extraction capabilities, M Block, to capture multi-scale information. To avoid overfitting caused by the small scale of the dataset, we propose a transfer learning method that can transfer priors experience from large computer vision datasets to multi-modality medical imaging datasets. Compared with other methods, our method outperforms other methods on HPC dataset, achieving the highest Dice of 82.98%. Our method is also superior to other methods on two public medical segmentation datasets, i.e., the CHASE_DB1 dataset and BraTS2018 dataset. On these two datasets, the Dice of our method is 79.83% and 84.87%, respectively. The code is available at: https://github.com/zhongqiu1245/TwistNet.

High-level ab initio mapping of the multiple H-abstraction pathways of the OH + glycine reaction.

We perform a systematic search in the transition-state (TS) and product-channel complex (MIN) regions of the multi-channel OH + glycine → H2O + H2N-CH-COOH (CH)/HN-CH2-COOH (NH)/H2N-CH2-COO (COOH) reactions. Geometry optimizations reveal {7, 3, 3} CH-TS, {2, 2, 2} CH-MIN, {17, 10, 5} NH-TS, {35, 19, 19} NH-MIN, and {6, 5, 5} COOH-TS conformers at the {MP2/3-21G, MP2/aug-cc-pVDZ, CCSD(T)-F12b/aug-cc-pVDZ} levels of theory as well as 2 additional CH-TSs based on chemical intuition. The benchmark relative energies of the TS, MIN, and product conformers are obtained by considering basis set effects up to aug-cc-pVQZ using the explicitly-correlated CCSD(T)-F12b method, post-(T) correlation up to CCSDT(Q), core correlation, scalar relativistic effects, spin-orbit coupling, and zero-point energy corrections. All the CH [ΔEe(ΔH0) = -38.54(-38.61) kcal mol-1], NH [ΔEe(ΔH0) = -16.72(-17.98) kcal mol-1], and COOH [ΔEe = -4.98 kcal mol-1] reactions are exothermic and proceed via shallow, usually negative, classical(adiabatic) barriers of -0.37(-0.95), -1.91(-2.48), and 1.02(-0.57) kcal mol-1, respectively. In the entrance channel MRCI/aug-cc-pVTZ computations reveal several complexes with reactive(non-reactive) arrangements and binding energies of 1.0, 1.6, 3.3, (5.2 and 5.9) kcal mol-1, stabilized by CH⋯OH, NH⋯OH, COOH⋯OH, (OH⋯OC and OH⋯N) hydrogen bonds, respectively.

Feature extraction semilearning and augmented representation for image captioning in crowd scenes

Image captioning has been an interesting task since 2015. The topic lies in the gap between Computer Vision and Natural Language Processing research directions. The problem can be described as follows: Given the input as a three-channel RGB image, a language model is trained to generate the hypothesis caption that describes the images’ contexts. In this study, we focus on solving image captioning in images captured in a crowd scene, which is more complicated and challenging. In general, a semilearning feature extraction mechanism is proposed to obtain more valuable high-level feature maps of images. Moreover, an augmented approach in the Transformer Encoder is explored to enhance the representation ability. The obtained results are promising and outperform those of other state-of-the-art captioning models on the CrowdCaption dataset.

Unsupervised Domain Adaptation Augmented by Mutually Boosted Attention for Semantic Segmentation of VHR Remote Sensing Images

This work investigates unsupervised domain adaptation (UDA)-based semantic segmentation of very high-resolution (VHR) remote sensing (RS) images from different domains. Most existing UDA methods resort to generative adversarial networks (GANs) to cope with the domain shift problem caused by the discrepancies across different domains. However, these GAN-based UDA methods directly align two domains in the appearance, latent, or output space based on convolutional neural networks (CNNs), making them ineffective in exploiting long-range dependencies across the high-level feature maps derived from different domains. Unfortunately, such high-level features play an essential role in characterizing RS images with complex content. To circumvent this obstacle, a mutually boosted attention transformer (MBATrans) is proposed to capture cross-domain dependencies of semantic feature representations in this work. Compared with conventional UDA methods, MBATrans can significantly reduce domain discrepancies by capturing transferable features using global attention. More specifically, MBATrans utilizes a novel mutually boosted attention (MBA) module to align cross-domain feature maps while enhancing domain-general features. Furthermore, a novel GAN-based network with improved discriminative capability is devised by integrating an additional discriminator to learn domain-specific features. Extensive experiments on two large-scale VHR RS datasets, namely, International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam and Vaihingen, confirm the superior performance of the proposed MBATrans-augmented GAN (MBATA-GAN) architecture. The source code in this work is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sstary/SSRS</uri> .

Multichannel Cross-Scale Semantic Coherent Attention Network for Image Inpainting

This paper investigates a cross-scale space semantic feature coherent image inpainting approach since it is challenging for the existing image inpainting methods to fuse the semantic feature information effectively. Firstly, we learn the feature semantic relevance step-by-step from the high-level semantic feature map’s attention mechanism and then we apply what we have learned to the preceding low-level feature map. In order to preserve the visual and semantic coherence of image repair, the missing content can be filled by changing attention from deep to shallow in a multiscale manner. A broader receptive field is generated by partial convolution, and semantic feature relevance is achieved using a multiscale cross feature space feature attention mechanism based on semantic attention. This technique improves the extensibility and continuity of the restored images by reconstructing the semantic information of different feature spaces, not only taking into account the reuse of existing semantic space features but also including across feature spaces. The experimental results demonstrated an improvement in PSNR, SSIM, and L1 performance by 10.50%, 0.13%, and 47.09%, respectively, with clear benefits.

Multi-Scale Residual Aggregation Feature Pyramid Network for Object Detection

The effective use of multi-scale features remains an open problem for object detection tasks. Recently, proposed object detectors have usually used Feature Pyramid Networks (FPN) to fuse multi-scale features. Since Feature Pyramid Networks use a relatively simple feature map fusion approach, it can lead to the loss or misalignment of semantic information in the fusion process. Several works have demonstrated that using a bottom-up structure in a Feature Pyramid Network can shorten the information path between lower layers and the topmost feature, allowing an adequate exchange of semantic information from different layers. We further enhance the bottom-up path by proposing a multi-scale residual aggregation Feature Pyramid Network (MSRA-FPN), which uses a unidirectional cross-layer residual module to aggregate features from multiple layers bottom-up in a triangular structure to the topmost layer. In addition, we introduce a Residual Squeeze and Excitation Module to mitigate the aliasing effects that occur when features from different layers are aggregated. MSRA-FPN enhances the semantic information of the high-level feature maps, mitigates the information decay during feature fusion, and enhances the detection capability of the model for large objects. It is experimentally demonstrated that our proposed MSRA-FPN improves the performance of the three baseline models by 0.5–1.9% on the PASCAL VOC dataset and is also quite competitive with other state-of-the-art FPN methods. On the MS COCO dataset, our proposed method can also improve the performance of the baseline model by 0.8% and the baseline model’s performance for large object detection by 1.8%. To further validate the effectiveness of MSRA-FPN for large object detection, we constructed the Thangka Figure Dataset and conducted comparative experiments. It is experimentally demonstrated that our proposed method improves the performance of the baseline model by 2.9–4.7% on this dataset and can reach up to 71.2%.

Concrete crack detection using lightweight attention feature fusion single shot multibox detector

As one of the most important defects of concrete, cracks seriously threaten the service life and safety of concrete structures, and various safety incidents caused by the collapse of concrete structures have occurred. Therefore, it is essential to detect concrete cracks as soon as possible. Existing object detection methods have low detection accuracy for cracks, leading to unsatisfactory detection results. In this paper, we propose a variety of feasible modules that improve the accuracy of single shot multibox detection (SSD), which is the most efficient object detection method in terms of both accuracy and speed. First, to improve the neural network’s ability to learn high-level and low-level feature maps, we propose a feature fusion enhancement module (FFEM). Second, to more effectively capture the information between feature map channels, we propose convolutional network attention (CNA). Third, to improve the anchor box fit to the ground truth box, we reset the distribution of the anchor box. Last, we propose a new type of nonmaximum suppression (NMS) named T-Soft NMS to address numerous issues with current NMS and to significantly enhance the performance of the model. We tested our method on a crack dataset, and numerous tests showed that it outperformed competing methods. In addition, we carried out ablation studies to confirm the validity and efficacy of our method.

Skin lesion classification based on two-modal images using a multi-scale fully-shared fusion network

Background and objectiveDue to the complexity of skin lesion features, computer-aided diagnosis of skin diseases based on multi-modal images is considered a challenging task. Dermoscopic images and clinical images are commonly used to diagnose skin diseases in clinical scenarios, and the complementarity of their features promotes the research of multi-modality classification in the computer-aided diagnosis field. Most current methods focus on the fusion between modalities and ignore the complementary information within each of them, which leads to the loss of the intra-modality relation. Multi-modality models for integrating features both within single modalities and across multiple modalities are limited in the literature. Therefore, a multi-modality model based on dermoscopic and clinical images is proposed to address this issue. MethodsWe propose a Multi-scale Fully-shared Fusion Network (MFF-Net) that gathers features of dermoscopic images and clinical images for skin lesion classification. In MFF-Net, the multi-scale fusion structure combines deep and shallow features within individual modalities to reduce the loss of spatial information in high-level feature maps. Then Dermo-Clinical Block (DCB) integrates the feature maps from dermoscopic images and clinical images through channel-wise concatenation and using a fully-shared fusion strategy that explores complementary information at different stages. ResultsWe validated our model on a four-class two-modal skin diseases dataset, and proved that the proposed multi-scale structure, the fusion module DCBs, and the fully-shared fusion strategy improve the performance of MFF-Net independently. Our method achieved the highest average accuracy of 72.9% on the 7-point checklist dataset, outperforming the state-of-the-art single-modality and multi-modality methods with an accuracy boost of 7.1% and 3.4%, respectively. ConclusionsThe multi-scale fusion structure demonstrates the significance of intra-modality relations between clinical images and dermoscopic images. The proposed network combined with the multi-scale structure, DCBs, and the fully-shared fusion strategy, can effectively integrate the features of the skin lesions across the two modalities and achieved a promising accuracy among different skin diseases.

An emotion analysis in learning environment based on theme-specified drawing by convolutional neural network.

Emotion in the learning process can directly influence the learner's attention, memory, and cognitive activities. Several literatures indicate that hand-drawn painting could reflect the learner's emotional status. But, such an evaluation of emotional status, manually conducted by the psychologist, is usually subjective and inefficient for clinical practice. To address the issues of subjectivity and inefficiency in the painting based emotional analysis, we conducted an exploration of a painting based emotional analysis in learning environment by using convolutional neural network model. A painting image of 100 × 100 pixels was used as input for the model. The instant emotional statue of the learner was collected by filling out a questionnaire and was reviewed by a psychologist and then used as the label for training the convolutional neural network model. With the completion of convolutional, full-connected, and classification operations, the features of the painting image were learned from the underlying pixel matrix to the high-level semantic feature mapping. Then the emotional classification of the painting image could be made to reflect the learner's emotional status. Finally, the classification result by the model was compared with the result manually conducted by a psychologist to validate the model accuracy. We conducted an experiment in a university at Hangzhou, and 2,103 learners joined in the experiment. The learner was required to first fill out a questionnaire reporting emotional status in the learning process, and then to complete a theme-specified painting. Two thousand valid paintings were received and divided into training dataset (1,600) and test dataset (400). The experimental result indicated that the model achieved the accuracy of 72.1%, which confirmed the effectiveness of the model for emotional analysis.

Question-guided feature pyramid network for medical visual question answering

Medical VQA (VQA-Med) is a critical multi-modal task that raises attention from the community. Existing models utilized just one high-level feature map (i.e., the last layer feature map) extracted by CNN and then fused it with semantic features through the co-attention mechanism. However, only using the high-level feature as a visual feature often ignores the details of the image, which are crucial for VQA-Med. In addition, questions often serve as a guide to targets of attention in the medical image. Therefore, in this paper, we propose a question-guided Feature Pyramid Network (QFPN) for VQA-Med. It extracts multi-level visual features with a feature pyramid network (FPN). In this way, the multi-scale information of medical images can be captured by using the high resolution of low-level features and rich semantic information of high-level features. Besides, a novel question-guided dynamic filter network (DFN) is designed to modulate the fusion progress of multi-level visual features and semantic features with respect to the raised question. Extensive results have demonstrated the effectiveness of the QFPN. Especially, we beat the winner of the 2019 ImageCLEF challenge and achieved 63.8% Accuracy and 65.7% BLEU in the ImageCLEF 2019 VQA-Med dataset.

Dense context distillation network for semantic parsing of oblique UAV images

Semantic segmentation of oblique unmanned aerial vehicle (UAV) images serves as a foundation for many modern urban applications, such as road scene monitoring and semantic 3D modeling. However, objects in UAV images can vary intensely in size and undergo severe perspective distortion because of the oblique viewing style. Existing general segmentation models designed for ground and remote sensing images rarely considered these challenges specific to UAV images. Therefore, they have large difficulties in learning discriminative representation for simultaneously reasoning the extremely large and small objects in UAV images. In this paper, we propose a dense context distillation network (DCDNet) to learn distortion-robust feature representation for semantic segmentation of UAV images. The basic DCDNet is deployed as an dual-branch encoder–decoder architecture. To accomplish the goal of dense context distillation, DCDNet is first equipped with several cross-scale context selectors at different encoding stages to densely and selectively gather the useful context from low- to high-level dual-scale feature maps. A joint supervision is then applied to reinforce the learning of shallower features for distilling more low-level contexts that are vital to the reasoning of small or thin structures. A multi-scale feature aggregator is incorporated to adaptively fuse the long-range context during decoding, which absorbs the complementary merits of the dense context captured from feature maps of different levels. With the dense context distillation, DCDNet is more capable of offering the differently scaled objects with the required context for better learning and prediction. Extensive experiments on the challenging UAVid dataset demonstrate that our DCDNet can well adapt to the oblique UAV images, achieving a state-of-the-art segmentation performance with a mIoU score of 72.38%. • A DCDNet is proposed for semantic parsing of UAV images. • DCDNet offers enough context for reasoning the large and small road scene objects. • DCDNet achieves state-of-the-art segmentation performance.