High-level Semantic Information Research Articles

A two‐scaled fully convolutional learning network for road detection

This paper aims to detect road regions based on a two-scaled deep neural network. The information from different scales is helpful to boost the performance of deep learning models, and it is also a widely used strategy in various computer vision applications. In the two-scaled model, skip-architecture and fully convolutional layers are used to fuse the low-level details and high-level semantic information. It enables to detect the road areas by multi-scale feature maps from different reception fields. To avoid the redundancy of scale information and the loss of features caused by the pooling layer, the feature maps before the first pooling layer are adopted in our model. By the convolutional kernels, our model can balance the information of two scales automatically. The loss function is also improved, in which the intersection over union (IoU) term is taken into account to guide the model to learn more features on the whole road regions. Comprehensive experiments on three benchmark datasets demonstrate that this approach can reach state-of-the-art performance.

Multi-U-Net: Residual Module under Multisensory Field and Attention Mechanism Based Optimized U-Net for VHR Image Semantic Segmentation.

As the acquisition of very high resolution (VHR) images becomes easier, the complex characteristics of VHR images pose new challenges to traditional machine learning semantic segmentation methods. As an excellent convolutional neural network (CNN) structure, U-Net does not require manual intervention, and its high-precision features are widely used in image interpretation. However, as an end-to-end fully convolutional network, U-Net has not explored enough information from the full scale, and there is still room for improvement. In this study, we constructed an effective network module: residual module under a multisensory field (RMMF) to extract multiscale features of target and an attention mechanism to optimize feature information. RMMF uses parallel convolutional layers to learn features of different scales in the network and adds shortcut connections between stacked layers to construct residual blocks, combining low-level detailed information with high-level semantic information. RMMF is universal and extensible. The convolutional layer in the U-Net network is replaced with RMMF to improve the network structure. Additionally, the multiscale convolutional network was tested using RMMF on the Gaofen-2 data set and Potsdam data sets. Experiments show that compared to other technologies, this method has better performance in airborne and spaceborne images.

A Multi-Task Sequential State Model for the Human Trajectory Data Understanding

Abstract Human trajectory data, collected from various location-based services, is of great significance to the understanding of users. However, trajectory-based user understanding is very challenging, due to the huge semantic gap between the existing low-level GEO spatial information and the target high-level semantic information. In this work, we propose a sequential state model as well as a multi-task based learning method to bridge the above semantic gap. First, we propose a sequential state model to organize the human trajectory data as well as the POI information. Second, we employ a LSTM based representation method to extract the semantic representation from the sequential state model, in which various representations are learned by using the user tags as the supervise information individually. Finally, we devise a multi-task fine-tuning LSTM method to take advantage of the dependency among the tags. We also demonstrate the usage of the proposed method and the effectiveness of the proposal in a real-world demand-side-platform system.

Multilevel feature fusion dilated convolutional network for semantic segmentation

Recently, convolutional neural network (CNN) has led to significant improvement in the field of computer vision, especially the improvement of the accuracy and speed of semantic segmentation tasks, which greatly improved robot scene perception. In this article, we propose a multilevel feature fusion dilated convolution network (Refine-DeepLab). By improving the space pyramid pooling structure, we propose a multiscale hybrid dilated convolution module, which captures the rich context information and effectively alleviates the contradiction between the receptive field size and the dilated convolution operation. At the same time, the high-level semantic information and low-level semantic information obtained through multi-level and multi-scale feature extraction can effectively improve the capture of global information and improve the performance of large-scale target segmentation. The encoder–decoder gradually recovers spatial information while capturing high-level semantic information, resulting in sharper object boundaries. Extensive experiments verify the effectiveness of our proposed Refine-DeepLab model, evaluate our approaches thoroughly on the PASCAL VOC 2012 data set without MS COCO data set pretraining, and achieve a state-of-art result of 81.73% mean interaction-over-union in the validate set.

Why can deep convolutional neural networks improve protein fold recognition? A visual explanation by interpretation.

As an essential task in protein structure and function prediction, protein fold recognition has attracted increasing attention. The majority of the existing machine learning-based protein fold recognition approaches strongly rely on handcrafted features, which depict the characteristics of different protein folds; however, effective feature extraction methods still represent the bottleneck for further performance improvement of protein fold recognition. As a powerful feature extractor, deep convolutional neural network (DCNN) can automatically extract discriminative features for fold recognition without human intervention, which has demonstrated an impressive performance on protein fold recognition. Despite the encouraging progress, DCNN often acts as a black box, and as such, it is challenging for users to understand what really happens in DCNN and why it works well for protein fold recognition. In this study, we explore the intrinsic mechanism of DCNN and explain why it works for protein fold recognition using a visual explanation technique. More specifically, we first trained a VGGNet-based DCNN model, termed VGGNet-FE, which can extract fold-specific features from the predicted protein residue-residue contact map for protein fold recognition. Subsequently, based on the trained VGGNet-FE, we implemented a new contact-assisted predictor, termed VGGfold, for protein fold recognition; we then visualized what features were extracted by each of the convolutional layers in VGGNet-FE using a deconvolution technique. Furthermore, we visualized the high-level semantic information, termed fold-discriminative region, of a predicted contact map from the localization map obtained from the last convolutional layer of VGGNet-FE. It is visually confirmed that VGGNet-FE could effectively extract distinct fold-discriminative regions for different types of protein folds, thereby accounting for the improved performance of VGGfold for protein fold recognition. In summary, this study is of great significance for both understanding the working principle of DCNNs in protein fold recognition and exploring the relationship between the predicted protein contact map and protein tertiary structure. This proposed visualization method is flexible and applicable to address other DCNN-based bioinformatics and computational biology questions. The online web server of VGGfold is freely available at http://csbio.njust.edu.cn/bioinf/vggfold/.

Multi-Scale Context-Guided Deep Network for Automated Lesion Segmentation With Endoscopy Images of Gastrointestinal Tract.

Accurate lesion segmentation based on endoscopy images is a fundamental task for the automated diagnosis of gastrointestinal tract (GI Tract) diseases. Previous studies usually use hand-crafted features for representing endoscopy images, while feature definition and lesion segmentation are treated as two standalone tasks. Due to the possible heterogeneity between features and segmentation models, these methods often result in sub-optimal performance. Several fully convolutional networks have been recently developed to jointly perform feature learning and model training for GI Tract disease diagnosis. However, they generally ignore local spatial details of endoscopy images, as down-sampling operations (e.g., pooling and convolutional striding) may result in irreversible loss of image spatial information. To this end, we propose a multi-scale context-guided deep network (MCNet) for end-to-end lesion segmentation of endoscopy images in GI Tract, where both global and local contexts are captured as guidance for model training. Specifically, one global subnetwork is designed to extract the global structure and high-level semantic context of each input image. Then we further design two cascaded local subnetworks based on output feature maps of the global subnetwork, aiming to capture both local appearance information and relatively high-level semantic information in a multi-scale manner. Those feature maps learned by three subnetworks are further fused for the subsequent task of lesion segmentation. We have evaluated the proposed MCNet on 1,310 endoscopy images from the public EndoVis-Ab and CVC-ClinicDB datasets for abnormal segmentation and polyp segmentation, respectively. Experimental results demonstrate that MCNet achieves [Formula: see text] and [Formula: see text] mean intersection over union (mIoU) on two datasets, respectively, outperforming several state-of-the-art approaches in automated lesion segmentation with endoscopy images of GI Tract.

Alignment Enhancement Network for Fine-grained Visual Categorization

Fine-grained visual categorization (FGVC) aims to automatically recognize objects from different sub-ordinate categories. Despite attracting considerable attention from both academia and industry, it remains a challenging task due to subtle visual differences among different classes. Cross-layer feature aggregation and cross-image pairwise learning become prevailing in improving the performance of FGVC by extracting discriminative class-specific features. However, they are still inefficient to fully use the cross-layer information based on the simple aggregation strategy, while existing pairwise learning methods also fail to explore long-range interactions between different images. To address these problems, we propose a novel Alignment Enhancement Network (AENet), including two-level alignments, Cross-layer Alignment (CLA) and Cross-image Alignment (CIA). The CLA module exploits the cross-layer relationship between low-level spatial information and high-level semantic information, which contributes to cross-layer feature aggregation to improve the capacity of feature representation for input images. The new CIA module is further introduced to produce the aligned feature map, which can enhance the relevant information as well as suppress the irrelevant information across the whole spatial region. Our method is based on an underlying assumption that the aligned feature map should be closer to the inputs of CIA when they belong to the same category. Accordingly, we establish Semantic Affinity Loss to supervise the feature alignment within each CIA block. Experimental results on four challenging datasets show that the proposed AENet achieves the state-of-the-art results over prior arts.

Top-Down Inference in the Auditory System: Potential Roles for Corticofugal Projections.

It has become widely accepted that humans use contextual information to infer the meaning of ambiguous acoustic signals. In speech, for example, high-level semantic, syntactic, or lexical information shape our understanding of a phoneme buried in noise. Most current theories to explain this phenomenon rely on hierarchical predictive coding models involving a set of Bayesian priors emanating from high-level brain regions (e.g., prefrontal cortex) that are used to influence processing at lower-levels of the cortical sensory hierarchy (e.g., auditory cortex). As such, virtually all proposed models to explain top-down facilitation are focused on intracortical connections, and consequently, subcortical nuclei have scarcely been discussed in this context. However, subcortical auditory nuclei receive massive, heterogeneous, and cascading descending projections at every level of the sensory hierarchy, and activation of these systems has been shown to improve speech recognition. It is not yet clear whether or how top-down modulation to resolve ambiguous sounds calls upon these corticofugal projections. Here, we review the literature on top-down modulation in the auditory system, primarily focused on humans and cortical imaging/recording methods, and attempt to relate these findings to a growing animal literature, which has primarily been focused on corticofugal projections. We argue that corticofugal pathways contain the requisite circuitry to implement predictive coding mechanisms to facilitate perception of complex sounds and that top-down modulation at early (i.e., subcortical) stages of processing complement modulation at later (i.e., cortical) stages of processing. Finally, we suggest experimental approaches for future studies on this topic.

Global context guided hierarchically residual feature refinement network for defocus blur detection

As an important pre-processing step, defocus blur detection makes critical role in various computer vision tasks. However, previous methods cannot obtain satisfactory results due to the complex image background clutter, scale sensitivity and miss of region boundary details. In this paper, for addressing these issues, we introduce a global context guided hierarchically residual feature refinement network (HRFRNet) for defocus blur detection from a natural image. In our network, the low-level fine detail features, high-level semantic and global context information are aggregated in a hierarchical manner to boost the final detection performance. In order to reduce the affect of complex background clutter and smooth regions without enough textures on the final results, we design a multi-scale dilation convolution based global context pooling module to capture the global context information from the most deep feature layer of the backbone feature extraction network. Then, a global context guiding module is introduced to add the global context information into different feature refining stages for guiding the feature refining process. In addition, by considering that the defocus blur is sensitive to image scales, we add a deep features guided fusion module to integrate the outputs of different stages for generating the final score map. Extensive experiments with ablation studies on two commonly used datasets are carried out to validate the superiority of our proposed network when compared with other 11 state-of-the-art methods in terms of both efficiency and accuracy.

Visual SLAM for robot navigation in healthcare facility

The COVID-19 pandemic has affected many countries, posing a threat to human health and safety, and putting tremendous pressure on the medical system. This paper proposes a novel SLAM technology using RGB and depth images to improve hospital operation efficiency, reduce the risk of doctor-patient cross-infection, and curb the spread of the COVID-19. Most current visual SLAM researches assume that the environment is stationary, which makes handling real-world scenarios such as hospitals a challenge. This paper proposes a method that effectively deals with SLAM problems for scenarios with dynamic objects, e.g., people and movable objects, based on the semantic descriptor extracted from images with help of a knowledge graph. Specifically, our method leverages a knowledge graph to construct a priori movement relationship between entities and establishes high-level semantic information. Built upon this knowledge graph, a semantic descriptor is constructed to describe the semantic information around key points, which is rotation-invariant and robust to illumination. The seamless integration of the knowledge graph and semantic descriptor helps eliminate the dynamic objects and improves the accuracy of tracking and positioning of robots in dynamic environments. Experiments are conducted using data acquired from healthcare facilities, and semantic maps are established to meet the needs of robots for delivering medical services. In addition, to compare with the state-of-the-art methods, a publicly available dataset is used in our evaluation. Compared with the state-of-the-art methods, our proposed method demonstrated great improvement with respect to both accuracy and robustness in dynamic environments. The computational efficiency is also competitive.

Bi-directional skip connection feature pyramid network and sub-pixel convolution for high-quality object detection

In existing state-of-the-art object detectors, feature pyramid networks (FPN) and multiscale feature fusion are still typically used. The traditional FPN fusion strategy is based on the top-down fusion of high-level semantic information. The top-down fusion method generally uses upsampling based on interpolation, which often results in jagged edges, mosaic distortion, and edge blurring. Moreover, in order to improve accuracy, the FPN-based fusion strategy must add multiple top-down components for fusion, which increases computational costs and leads to a poor balance between precision and speed. In this paper, we propose a novel fusion strategy based on a backbone network. We aim to design simple and efficient components for high-quality object detection. Our proposed strategy, bi-directional skip connection FPN (BiSCFPN), consists of three components: a bi-directional skip connection (BiSC), a selective dilated convolution module (SDCM), and sub-pixel convolution (SP). The BiSC aims to enhance semantic information between different feature layers in the backbone network and simultaneously uses the SDCM to improve the receptive fields of differently sized targets in the fusion stage. Finally, SP learns the relationship between the features of upsampling and downsampling images to effectively mitigate the problems caused by the traditional interpolation method. BiSCFPN achieves an average precision of 38.2% in tests with the Microsoft Common Objects in Context (MS COCO) test-dev dataset at a real-time speed of ~50 FPS (608×608) using an Nvidia GeForce RTX 2080 Ti graphics card and significantly improves the balance between precision and speed.

Multi-scale retinal vessel segmentation using encoder-decoder network with squeeze-and-excitation connection and atrous spatial pyramid pooling.

The segmentation of blood vessels in retinal images is crucial to the diagnosis of many diseases. We propose a deep learning method for vessel segmentation based on an encoder-decoder network combined with squeeze-and-excitation connection and atrous spatial pyramid pooling. In our implementation, the atrous spatial pyramid pooling allows the network to capture features at multiple scales, and the high-level semantic information is combined with low-level features through the encoder-decoder architecture to generate segmentations. Meanwhile, the squeeze-and-excitation connections in the proposed network can adaptively recalibrate features according to the relationship between different channels of features. The proposed network can achieve precise segmentation of retinal vessels without hand-crafted features or specific post-processing. The performance of our model is evaluated in terms of visual effects and quantitative evaluation metrics on two publicly available datasets of retinal images, the Digital Retinal Images for Vessel Extraction and Structured Analysis of the Retina datasets, with comparison to 12 representative methods. Furthermore, the proposed network is applied to vessel segmentation on local retinal images, which demonstrates promising application prospect in medical practices.

Locality-Aware Rotated Ship Detection in High-Resolution Remote Sensing Imagery Based on Multiscale Convolutional Network

Ship detection has been an active and vital topic in the field of remote sensing for a decade, but it is still a challenging problem due to the large-scale variations, the high aspect ratios, the intensive and rotated arrangement, and the background clutter disturbance. In this letter, we propose a locality-aware rotated ship detection (LARSD) framework based on a multiscale convolutional neural network (CNN) to tackle these issues. The proposed framework applies a UNet-like multiscale CNN to generate multiscale feature maps with high-level semantic information in high resolution. Then, an anchor-based rotated bounding box regression is applied for directly predicting the probability, the edge distances, and the angle of ships. Finally, a locality-aware score alignment (LASA) is proposed to fix the mismatch between classification results and location results caused by the independence of each subnet. Furthermore, to enlarge the data sets of ship detection, we build a new high-resolution ship detection (HRSD) data set, where 2499 images and 9269 instances were collected from Google Earth with different resolutions. Experiments based on public data set high-resolution ship collection 2016 (HRSC2016) and our HRSD data set demonstrate that our detection method achieves the state-of-the-art performance.

Vehicle detection algorithm based on LW-SSD

Due to the complex structure of the traditional SSD network model and the large amount of parameters, it is not suitable for porting to mobile or embedded platforms, so the LW-SSD vehicle detection algorithm is proposed, this paper redesigns the basic feature extraction network of SSD, using MobileNet feature extraction network instead of VGG16, for the original MobileNet to improve, remove the original MobileNet network average pooling layer and Softmax layer, while reducing the MobileNet network from the original 28 layers to 13 layers. Then, on the basis of the subsequent network, feature maps of different scales are scaled, residual mapping and deep feature fusion operations, and the low-level position information and high-level semantic information are effectively combined to obtain new feature maps of different scales. The designed network model is trained using the Tensorflow framework, and the test effect of the network model is evaluated. The experimental results show that the accuracy of the network model designed in this paper is 78.76%, which is 9.27% lower than the original SSD network model. However, the amount of parameters is only 1/4 of the SSD network model. At the expense of partial accuracy, the design of a lightweight network model is exchanged, which lays the foundation for subsequent development on mobile or embedded platforms.

Rethinking Image Salient Object Detection: Object-Level Semantic Saliency Reranking First, Pixelwise Saliency Refinement Later.

Human attention is an interactive activity between our visual system and our brain, using both low-level visual stimulus and high-level semantic information. Previous image salient object detection (SOD) studies conduct their saliency predictions via a multitask methodology in which pixelwise saliency regression and segmentation-like saliency refinement are conducted simultaneously. However, this multitask methodology has one critical limitation: the semantic information embedded in feature backbones might be degenerated during the training process. Our visual attention is determined mainly by semantic information, which is evidenced by our tendency to pay more attention to semantically salient regions even if these regions are not the most perceptually salient at first glance. This fact clearly contradicts the widely used multitask methodology mentioned above. To address this issue, this paper divides the SOD problem into two sequential steps. First, we devise a lightweight, weakly supervised deep network to coarsely locate the semantically salient regions. Next, as a postprocessing refinement, we selectively fuse multiple off-the-shelf deep models on the semantically salient regions identified by the previous step to formulate a pixelwise saliency map. Compared with the state-of-the-art (SOTA) models that focus on learning the pixelwise saliency in single images using only perceptual clues, our method aims at investigating the object-level semantic ranks between multiple images, of which the methodology is more consistent with the human attention mechanism. Our method is simple yet effective, and it is the first attempt to consider salient object detection as mainly an object-level semantic reranking problem.

Defocus Blur Detection via Boosting Diversity of Deep Ensemble Networks.

Existing defocus blur detection (DBD) methods usually explore multi-scale and multi-level features to improve performance. However, defocus blur regions normally have incomplete semantic information, which will reduce DBD's performance if it can't be used properly. In this paper, we address the above problem by exploring deep ensemble networks, where we boost diversity of defocus blur detectors to force the network to generate diverse results that some rely more on high-level semantic information while some ones rely more on low-level information. Then, diverse result ensemble makes detection errors cancel out each other. Specifically, we propose two deep ensemble networks (e.g., adaptive ensemble network (AENet) and encoder-feature ensemble network (EFENet)), which focus on boosting diversity while costing less computation. AENet constructs different light-weight sequential adapters for one backbone network to generate diverse results without introducing too many parameters and computation. AENet is optimized only by the self- negative correlation loss. On the other hand, we propose EFENet by exploring the diversity of multiple encoded features and ensemble strategies of features (e.g., group-channel uniformly weighted average ensemble and self-gate weighted ensemble). Diversity is represented by encoded features with less parameters, and a simple mean squared error loss can achieve the superior performance. Experimental results demonstrate the superiority over the state-of-the-arts in terms of accuracy and speed. Codes and models are available at: https://github.com/wdzhao123/DENets.

Graph Regularized Hierarchical Diffusion Process With Relevance Feedback for Medical Image Retrieval

Befitting from the interpretability and the capacity in capturing the underlying manifold structure, diffusion process (DP) has attracted increasing attention in the field of image retrieval. Within it, hierarchical diffusion process (HDP) has achieved satisfactory results in retrieved performance and complexity. However, the existing hierarchical diffusion process methods only diffuse the affinity values in low-level visual space without considering the high-level semantic information, which cause the problem of semantic gap. To overcome these problems, we propose a Graph Regularized Hierarchical Diffusion Process (GRHDP) method with relevance feedback, and apply it to retrieve medical images. The proposed algorithm firstly establishes a hierarchical structure of the images in medical image database and spreads the affinity values among query images and top-layer images by graph regularization diffusion. Then relevance feedback is introduced to adjust the similarity between query images and retrieved images in top layer, and the affinity values are diffused again according to labeled information of feedback. Finally, the similarity between queries and others in database can be obtained by interpolating the diffused results on the top layer from top to bottom. The experimental results show that our proposed GRHDP with relevance feedback has achieved better retrieval performance than manifold ranking and regularized diffusion process (RDP) when returning top retrieved images.

Cascaded Segmented Matting Network for Human Matting

Human matting, high quality extraction of humans from natural images, is crucial for a wide variety of applications such as virtual reality, augmented reality, entertainment and so on. Since the matting problem is an ill-posed problem, most previous methods rely on extra user inputs such as trimap or scribbles as guidance to estimate alpha value for the pixels that are in the unknown region of the trimap. This phenomenon makes it difficult to be applied to large scale data. In order to solve these problems, we studied the unique role of semantics and details in image matting, and decomposed the matting task into two sub-tasks: trimap segmentation based on high-level semantic information and alpha regression based on low-level detailed information. Specifically, we proposed a novel Cascaded Segmented Matting Network (CSMNet), which uses a shared encoder and two separate decoders to learn these two tasks in a collaborative way to achieve the end-to-end human image matting. In addition, we established a large-scale dataset with 14,000 fine-labeled human matting images. A background dataset is also built to simulate real pictures. Comprehensive empirical studies on above datasets demonstrate that CSMNet could produce a stable and accurate alpha matte without the input of trimap and achieve an evaluation value that is comparable to the algorithm that requires trimap.

A Real-Time Integrated Face Mask Detector to Curtail Spread of Coronavirus

Effective strategies to control COVID-19 pandemic need high attention to mitigate negatively impacted communal health and global economy, with the brim-full horizon yet to unfold. In the absence of effective antiviral and limited medical resources, many measures are recommended by WHO to control the infection rate and avoid exhausting the limited medical resources. Wearing mask is among the non-pharmaceutical intervention measures that can be used as barrier to primary route of SARS-CoV2 droplets expelled by presymptomatic or asymptomatic individuals. Regardless of discourse on medical resources and diversities in masks, all countries are mandating coverings over nose and mouth in public areas. Towards contribution of public health, the aim of the paper is to devise a real-time technique that can efficiently detect non mask faces in public and thus enforce to wear mask. The proposed technique is ensemble of one stage and two stage detectors to achieve low inference time and high accuracy. We took ResNet50 as a baseline model and applied the concept of transfer learning to fuse high level semantic information in multiple feature maps. In addition, we also propose a bounding box transformation to improve localization performance during mask detection. The experiments are conducted with three popular baseline models namely ResNet50, AlexNet and MobileNet. We explored the possibility of these models to plug-in with the proposed model, so that highly accurate results can be achieved in less inference time. It is observed that the proposed technique can achieve high accuracy (98.2%) when implemented with ResNet50. Besides, the proposed model can generate 11.07% and 6.44% higher precision and recall respectively in mask detection when compared to RetinaFaceMask detector.

IT-HYFAO-VO: Interpretation Tree-Based VO With Hybrid Feature Association and Optimization

For the visual odometry (VO) and simultaneous localization and mapping (SLAM) in indoor environments, high-level geometric features have attracted more and more attention in recent years. Unlike the generally used point features, the geometric features, such as planes and lines, encode more higher-level semantic information of the scene which is beneficial for various tasks of mobile robots. In this article, an RGB-D VO system with hybrid high-level geometric features is developed. An interpretation tree (IT)-based hybrid feature association framework is proposed, which turns the feature association into a multiple-hypothesis decision problem. The IT expansion method is elaborately designed. Specifically, an internode consistency is proposed for generation of hypotheses and a consistent transformation model (CTM) for each hypothesis is explicitly expressed and incrementally updated. When the IT is constructed, a closed-form solution to the feature association and the camera transformation can be obtained. Then, a hybrid feature joint optimization method is introduced to further refine the pose estimate and parameters of geometric features. During optimization, the planes and lines are appropriately parameterized and the uncertainties arising from feature extraction are derived and used to balance the contributions of two types of features in the cost function. Extensive experiments are executed on public datasets and the results demonstrate that the proposed method can achieve higher accuracy and stronger robustness.