High-level Semantics Research Articles

Audio Matters in Video Super-Resolution by Implicit Semantic Guidance

Video super-resolution (VSR) aims to use multiple consecutive low-resolution frames to recover the corresponding high-resolution frames. However, existing VSR methods only consider videos as image sequences, ignoring another essential timing informationaudio, while in fact, there is a semantic link between audio and vision, and extensive studies have shown that audio can provide supervisory information in visual networks. Meanwhile, the addition of semantic priors has been proven to be effective in super-resolution (SR) tasks, but a pretrained segmentation network is required to obtain semantic segmentation maps. By contrast, audio as the information contained in the video itself can be directly used. Therefore, in this study, we propose a novel and pluggable multiscale audiovisual fusion (MS-AVF) module to enhance VSR performance by exploiting the relevant audio information, which can be regarded as implicit semantic guidance compared with the kind of explicit segmentation priors. Specifically, we first fuse audiovisual features on the semantic feature maps of different granularities of the target frames, and then through a top-down multiscale fusion approach, feedback high-level semantics to the underlying global visual features layer by layer, thereby providing effective audio implicit semantic guidance for VSR. Experimental results show that audio can further improve the VSR effect. Moreover, by visualizing the learned attention mask, the proposed end-to-end model can automatically learn potential audiovisual semantic links, especially improving the accuracy and effectiveness of the SR of sound sources and their surrounding regions.

SAR Target Classification Based on Multiscale Attention Super-Class Network

The convolutional neural network (CNN) is widely used in synthetic aperture radar (SAR) target recognition, but conventional CNN mainly adopts a single-scale convolutional kernel, resulting in losing part of the feature information of targets and does not pay enough attention to significant features. On the other hand, conventional CNN approaches only assign fine-class labels to SAR targets, ignoring the high-level semantics information of similar categories, which reduces the feature differences between categories and the generalization ability of the model. Therefore, this paper proposes a multi-scale attention super-class CNN (MSA-SCNN) for SAR target classification. Firstly, MSA-SCNN combines multi-scale feature fusion with the attention module to improve the integrity of SAR target feature representation. The attention module includes channel and spatial attention modules, which realize the weighted enhancement of different scale features. Additionally, MSA-SCNN introduces super-class labels to increase the feature difference between categories. The classification stage consists of a fine-class branch and a super-class branch, and the features trained on the super-class branch are fused to the fine-class branch to improve the network's fine classification ability. Experiments on the moving and stationary target acquisition and recognition (MSTAR) dataset and the FUSAR-Ship dataset show that the proposed MSA-SCNN outperforms many current existing state-of-the-art methods.

MINet: Multilevel Inheritance Network-Based Aerial Scene Classification

Scene classification of aerial images is the basis of automatic recognition of complex scenes, and it is also a challenging computer vision task. In recent years, with the rapid development of deep learning, the semantic feature extraction method based on a convolutional neural network (CNN) has made great progress. Moreover, a recent study indicates that combining the semantic information of deep-layer features with the detailed texture information of shallow-layer features in CNN can further improve the performance of classification. In this letter, an end-to-end multilevel feature-based network named multilevel inheritance network (MINet) is proposed for aerial scene classification. First, the feature extraction module based on the feature pyramid network (FPN) is used to get multilevel feature maps. In the process of merging shallow features, high-level semantics of deep-layer are inherited. Then, an attention mechanism is added after the multilevel features to reduce the interference of redundant information and noise. Finally, we use a feature fusion module to automatically learn the weight of each feature layer and make a comprehensive decision. The effectiveness of the proposed method is verified in AID, WHU-RS19 and NWPU-RESISC45 datasets. Results show that the proposed method achieves competitive classification accuracy.

Automatic Graph Learning Convolutional Networks for Hyperspectral Image Classification

The excellent performance of graph convolutional networks (GCNs) on non-Euclidean data has drawn widespread attention from the hyperspectral image classification (HSIC) community, where the predefined graph (including node modeling and adjacency matrix calculation) plays a key role. However, existing GCN-based methods rely on manual efforts in constructing and updating graphs, and the superpixel-based node features lack high-level semantics. In this article, we propose an automatic graph learning convolutional network (Auto-GCN), which unifies the graph learning and HSIC in a “network-in-network” manner. Specifically, the graph is employed to model the interaction of the high-order tensors. Considering the powerful learning and representation capabilities of convolutional neural networks (CNNs), the semisupervised Siamese network (SiamNet) is embedded into GCNs and HSIC networks to accomplish the automatic learning and dynamic updating of the graph. GCNs further encode and infer the dynamic graph, and then, the learnable graph reprojection matrix is designed to assign graph representations to pixels. The dynamic graph serves the HSIC task during forward propagation, while the HSIC task continuously corrects the graph during backward propagation. Therefore, the “automatic” of the proposed Auto-GCN is not only reflected in the fact that the graph representation is designed and updated by an end-to-end network but is also HSIC task-oriented. The experimental results show that the proposed Auto-GCN outperforms other state-of-the-art methods on four publicly available hyperspectral datasets.

Image Searching System

Finding similar images on a visual sample is a difficult AI task, to solve which many works are devoted. The problem is to determine the essential properties of images of low and higher semantic level. Based on them, a vector of features is built, which will be used in the future to compare pairs of images. Each pair always includes an image from the collection and a sample image that the user is looking for. The result of the comparison is a quantity called the visual relativity of the images. Image properties are called features and are evaluated by calculation algorithms. Image features can be divided into low-level and high-level. Low-level features include basic colors, textures, shapes, significant elements of the whole image. These features are used as part of more complex recognition tasks. The main progress is in the definition of high-level features, which is associated with understanding the content of images. In this paper, research of modern algorithms is done for finding similar images in large multimedia databases. The main problems of determining high-level image features, algorithms of overcoming them and application of effective algorithms are described. The algorithms used to quickly determine the semantic content and improve the search accuracy of similar images are presented. The aim: The purpose of work is to conduct comparative analysis of modern image retrieval algorithms and retrieve its weakness and strength.

A Residual Attention and Local Context-Aware Network for Road Extraction from High-Resolution Remote Sensing Imagery

Extracting road information from high-resolution remote sensing images (HRI) can provide crucial geographic information for many applications. With the improvement of remote sensing image resolution, the image data contain more abundant feature information. However, this phenomenon also enhances the spatial heterogeneity between different types of roads, making it difficult to accurately discern the road and non-road regions using only spectral characteristics. To remedy the above issues, a novel residual attention and local context-aware network (RALC-Net) is proposed for extracting a complete and continuous road network from HRI. RALC-Net utilizes a dual-encoder structure to improve the feature extraction capability of the network, whose two different branches take different feature information as input data. Specifically, we construct the residual attention module using the residual connection that can integrate spatial context information and the attention mechanism, highlighting local semantics to extract local feature information of roads. The residual attention module combines the characteristics of both the residual connection and the attention mechanism to retain complete road edge information, highlight essential semantics, and enhance the generalization capability of the network model. In addition, the multi-scale dilated convolution module is used to extract multi-scale spatial receptive fields to improve the model’s performance further. We perform experiments to verify the performance of each component of RALC-Net through the ablation study. By combining low-level features with high-level semantics, we extract road information and make comparisons with other state-of-the-art models. The experimental results show that the proposed RALC-Net has excellent feature representation ability and robust generalizability, and can extract complete road information from a complex environment.

An Improved Kidney Tumor Prediction Using Deep Convolutional Neural Network-Restricted Boltzmann Machine Technique in Medical Image Segmentation

Automatic medical image segmentation has become increasingly important as contemporary medical imaging has become more widely available and used. Existing image segmentation solutions however lack the necessary functionality for simple medical image segmentation pipeline design. Pipelines that have already been deployed are frequently standalone software that has been optimised for a certain public data collection. As a result, the open-source python module deep-Convolutional neural network-Restricted Boltzmann Machine (deep CNNRBM) was introduced in this research work. The goal of Deep CNN-purpose RBMs is to have an easy-touse API that allows for the rapid creation of medical image segmentation transmission lines that include data augmentation, metrics, data I/O pre-processing, patch wise analysis, a library of pre-built deep neural networks, and fully automated assessment. Similarly, comprehensive pipeline customisation is possible because of strong configurability and many open interfaces. The dataset of Kidney tumor Segmentation challenge 2019 (KiTS19) acquired a strong predictor with respect to the standard 3D U-net model after cross-validation using deep CNNRBM. To that purpose, deep CNN-RBM, an expressive deep learning medical image segmentation architecture is introduced. The CNN sub-model captures frame-level spatial features automatically while the RBM submodel fuses spatial data over time to learn higher-level semantics in kidney tumor prediction. A neural network recognises medical picture segmentation, which is initiated using RBM to second-order collected data and then fine-tuned using back propagation to be more differential. According to the simulation outcome, the proposed deep CNN-RBM produced good classification results on the kidney tumour segmentation dataset.

Semantic Edge Detection with Diverse Deep Supervision

Semantic edge detection (SED), which aims at jointly extracting edges as well as their category information, has far-reaching applications in domains such as semantic segmentation, object proposal generation, and object recognition. SED naturally requires achieving two distinct supervision targets: locating fine detailed edges and identifying high-level semantics. Our motivation comes from the hypothesis that such distinct targets prevent state-of-the-art SED methods from effectively using deep supervision to improve results. To this end, we propose a novel fully convolutional neural network using diverse deep supervision within a multi-task framework where bottom layers aim at generating category-agnostic edges, while top layers are responsible for the detection of category-aware semantic edges. To overcome the hypothesized supervision challenge, a novel information converter unit is introduced, whose effectiveness has been extensively evaluated on SBD and Cityscapes datasets.

TDCMR: Triplet-Based Deep Cross-Modal Retrieval for Geo-Multimedia Data

Mass multimedia data with geographical information (geo-multimedia) are collected and stored on the Internet due to the wide application of location-based services (LBS). How to find the high-level semantic relationship between geo-multimedia data and construct efficient index is crucial for large-scale geo-multimedia retrieval. To combat this challenge, the paper proposes a deep cross-modal hashing framework for geo-multimedia retrieval, termed as Triplet-based Deep Cross-Modal Retrieval (TDCMR), which utilizes deep neural network and an enhanced triplet constraint to capture high-level semantics. Besides, a novel hybrid index, called TH-Quadtree, is developed by combining cross-modal binary hash codes and quadtree to support high-performance search. Extensive experiments are conducted on three common used benchmarks, and the results show the superior performance of the proposed method.

High-Resolution Boundary Refined Convolutional Neural Network for Automatic Agricultural Greenhouses Extraction from GaoFen-2 Satellite Imageries

Agricultural greenhouses (AGs) are an important component of modern facility agriculture, and accurately mapping and dynamically monitoring their distribution are necessary for agricultural scientific management and planning. Semantic segmentation can be adopted for AG extraction from remote sensing images. However, the feature maps obtained by traditional deep convolutional neural network (DCNN)-based segmentation algorithms blur spatial details and insufficient attention is usually paid to contextual representation. Meanwhile, the maintenance of the original morphological characteristics, especially the boundaries, is still a challenge for precise identification of AGs. To alleviate these problems, this paper proposes a novel network called high-resolution boundary refined network (HBRNet). In this method, we design a new backbone with multiple paths based on HRNetV2 aiming to preserve high spatial resolution and improve feature extraction capability, in which the Pyramid Cross Channel Attention (PCCA) module is embedded to residual blocks to strengthen the interaction of multiscale information. Moreover, the Spatial Enhancement (SE) module is employed to integrate the contextual information of different scales. In addition, we introduce the Spatial Gradient Variation (SGV) unit in the Boundary Refined (BR) module to couple the segmentation task and boundary learning task, so that they can share latent high-level semantics and interact with each other, and combine this with the joint loss to refine the boundary. In our study, GaoFen-2 remote sensing images in Shouguang City, Shandong Province, China are selected to make the AG dataset. The experimental results show that HBRNet demonstrates a significant improvement in segmentation performance up to an IoU score of 94.89%, implying that this approach has advantages and potential for precise identification of AGs.

A Modified HSIFT Descriptor for Medical Image Classification of Anatomy Objects

Modeling low level features to high level semantics in medical imaging is an important aspect in filtering anatomy objects. Bag of Visual Words (BOVW) representations have been proven effective to model these low level features to mid level representations. Convolutional neural nets are learning systems that can automatically extract high-quality representations from raw images. However, their deployment in the medical field is still a bit challenging due to the lack of training data. In this paper, learned features that are obtained by training convolutional neural networks are compared with our proposed hand-crafted HSIFT features. The HSIFT feature is a symmetric fusion of a Harris corner detector and the Scale Invariance Transform process (SIFT) with BOVW representation. The SIFT process is enhanced as well as the classification technique by adopting bagging with a surrogate split method. Quantitative evaluation shows that our proposed hand-crafted HSIFT feature outperforms the learned features from convolutional neural networks in discriminating anatomy image classes.

Modeling feedback representations in ventral visual cortex using a generative adversarial autoencoder

In less than the blink of an eye, the human brain processes visual sensory input, interprets the visual scene, identifies faces, and recognizes objects. Decades of neurophysiological studies have demonstrated that the brain accomplishes these complicated tasks through a dense network of feedforward and feedback neural processes in the ventral visual cortex. So far, these visual processes are primarily modeled with feedforward hierarchical neural networks, and the computational role of feedback processes is poorly understood. In this study, we developed a generative autoencoder neural network model and adversarially trained it on a large categorically diverse data set of images (Objects, scenes, faces, and animates). We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. To test the hypothesis, we compared representational similarity of the activity patterns in the internal layers of the proposed model with magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) data acquired while participants (N=15) viewed a set of 156 images organized in four categories of objects, scenes, faces, and animates. Our proposed model identified two segregated neural dynamics in the ventral visual pathway. The representational comparison with MEG data revealed a temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally subsequent dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Further, representational comparison of model encoder and decoder layers with two fMRI regions of interests, namely early visual cortex (EVC) and inferior temporal area (IT), revealed a growing categorical representation (similar to IT) along the encoder layers (feedforward sweep) and a progression in detail visual representations (akin to EVC) along the decoder layers (feedback sweep).

Exploring spatiotemporal changes in cities and villages through remote sensing using multibranch networks

With the rapid development of the social economy, monumental changes have taken place in the urban and rural environments. Urban and rural areas play a vital role in the interactions between humans and society. Traditional machine learning methods are used to perceive the massive changes in the urban and rural areas, though it is easy to overlook the detailed information about the changes made to the intentional target. As a result, the perception accuracy needs to be improved. Therefore, based on a deep neural network, this paper proposes a method to perceive the spatiotemporal changes in urban and rural intentional connotations through the perspective of remote sensing. The framework first uses multibranch DenseNet to model the multiscale spatiotemporal information of the intentional target and realizes the interaction of high-level semantics and low-level details in the physical appearance. Second, a multibranch and cross-channel attention module is designed to refine and converge multilevel and multiscale temporal and spatial semantics to perceive the subtle changes in the urban and rural intentional targets through the semantics and physical appearance. Finally, the experimental results show that the multibranch perception framework proposed in this paper has the best performance on the two baseline datasets A and B, and its F-Score values are 88.04% and 53.72%, respectively.

Terra: A Smart and Sensible Digital Twin Framework for Robust Robot Deployment in Challenging Environments

Digital twin (DT) systems that replicate the physical world digitally are powerful tools for monitoring physical systems and evaluating algorithms, but current DT systems are commonly not applicable for robotic deployment and investigation. Meanwhile, current 3-D simulation-based robotic platforms do not model the dynamics of the physical world on-the-fly as done in DT systems, limiting their potential for the development of robotics in challenging environments. To tackle this issue, we propose the first robot-centered smart DT framework, namely, Terra, to facilitate the deployment of robots in challenging environments. The proposed Terra framework introduces a comprehensive DT representation to encode the useful real-time dynamics of both the physical world and the robot agent deployed therein. A multiview multimodality perception module is further devised for Terra to obtain high-level semantics and deliver a precise description of the current status of the environment and the robot agent. By mapping the perceived results to the virtual replica of the physical environment, Terra actively updates the action policy and sends it back to the agent, forming an integral and real-time information feedback loop. In practice, to help demonstrate the effectiveness and feasibility of the proposed framework, we deliberately set up a challenging unordered physical environment with many obstacles and a very simple robot aiming to fulfill a navigation task. Empirical results show that the proposed Terra framework successfully facilitates the robot to accomplish the task without causing hazards.

BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation

The low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, which leads to a considerable accuracy decrease. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for realtime semantic segmentation. To this end, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context. The Semantic Branch is lightweight due to reducing the channel capacity and a fast-downsampling strategy. Furthermore, we design a Guided Aggregation Layer to enhance mutual connections and fuse both types of feature representation. Besides, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture performs favourably against a few state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2,048x1,024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy.

Deep Siamese Networks Based Change Detection with Remote Sensing Images

Although considerable success has been achieved in change detection on optical remote sensing images, accurate detection of specific changes is still challenging. Due to the diversity and complexity of the ground surface changes and the increasing demand for detecting changes that require high-level semantics, we have to resort to deep learning techniques to extract the intrinsic representations of changed areas. However, one key problem for developing deep learning metho for detecting specific change areas is the limitation of annotated data. In this paper, we collect a change detection dataset with 862 labeled image pairs, where the urban construction-related changes are labeled. Further, we propose a supervised change detection method based on a deep siamese semantic segmentation network to handle the proposed data effectively. The novelty of the method is that the proposed siamese network treats the change detection problem as a binary semantic segmentation task and learns to extract features from the image pairs directly. The siamese architecture as well as the elaborately designed semantic segmentation networks significantly improve the performance on change detection tasks. Experimental results demonstrate the promising performance of the proposed network compared to existing approaches.

MASG-GAN: A multi-view attention superpixel-guided generative adversarial network for efficient and simultaneous histopathology image segmentation and classification

Efficient analysis of Haematoxylin and Eosin stained histopathology images has become a challenge in digital pathology work-flow. We propose a Multi-view Attention Superpixel-guided Generative Adversarial Network (MASG-GAN) to achieve the multi-task learning for nuclei segmentation and benign-malignant tissue classification. Firstly, a novel superpixel segmentation approach driven by Bounded Asymmetric Gaussian Mixture Model (BAGMM) is presented for generating superpixel-prior probability map with high-level semantics. Then, we develop a generator network that integrates student-branch and teacher-branch. Concretely, the teacher-branch takes superpixel-prior probability map as input and guides the student-branch for accurate segmentation and classification. Then in generator, we build a light-weight U-shaped block (LUB) that consists of depthwise separable convolutions with mini encoding-decoding structure to reduce network computational cost. Finally, a Multi-view Attention Module (MVAM) is designed for further enhance the segmentation quality of nuclei with small area and unclear boundary. Extensive experiments on five benchmark datasets demonstrate that our pipeline outperforms some state-of-the-art methods, especially in terms of efficiency.

Knowledge Acquisition from Pedestrian Flow Analysis using Sparse Mobile Probe Data

Autonomous vehicles require high-level semantic maps, which contain the activities of pedestrians and cars, to ensure safe navigation. High-level semantics can be obtained from mobile probe sensor data. Analyzing pedestrian trajectories obtained from mobile probe data is an effective approach to avoid collisions between autonomous vehicles and pedestrians. Such analyses of pedestrian trajectories can generate new information such as pedestrian behaviors in violation of traffic regulations. However, pedestrian trajectories obtained from mobile probe data significantly sparse and noisy, making it challenging to analyze pedestrian activity. To address this issue, we propose multiple daily data and graph-based approaches to treat sparse and noisy data for estimating the flow of pedestrians based on mobile probe data. To improve the sparseness of the data, multiple daily data are fused. After that, a pedestrian graph is created to enhance the region’s coverage by connecting the sparse data indicating the flow of pedestrians. This proposed approach successfully obtained pedestrian trajectory data from the sparse and noisy data. Moreover, it was possible to identify the potential locations where pedestrians tend to cross the street by analyzing the pedestrian flow. The results indicate that 83% of well-known regions where pedestrians tend to cross the street corresponded with those extracted using the proposed approach. Furthermore, a high-level semantic map of the regions where pedestrians tend to cross the street along a 1-km road is presented. The trajectory information obtained using the proposed approach is expected to be essential for understanding different scenarios of the interactions between individuals and autonomous vehicles.

Residual visualization-guided explainable copy-relationship learning for image copy detection in social networks

In modern times, people are used to sharing individual works, giving opinions, and expressing emotions and sentiments by transmitting a variety of images such as personal photographs, pictures of purchased products, and emoticons on social networks. However, this results in a lot of illegal image copies being distributed, which causes the issues of misleading the public and copyright infringement. To identify such images, recent copy detection approaches learn similarity functions by directly exploring neural networks such as Siamese and pseudo-Siamese networks and thereby determine the copy relationships between images. Unfortunately, since such approaches directly use these networks as black boxes, without considering the essential differences between image copies and similar images, handling the prevalent need for copy detection, i.e., identifying image copies from similar images, remains a challenge. To address the above issue, first, a residual visualization scheme is proposed in this study to visualize the residual image by subtracting the original image from the geometrically aligned test image, thereby revealing the relationship between the two images. Then, guided by the observations on the residual visualization results, a newly designed explainable copy-relationship learning network (ECRLN) architecture is presented. In this network architecture, the characteristics of residual images, ranging from local details to global appearances and from low-level visual patterns to high-level semantics, are sufficiently utilized to learn a measurement function for computing the copy-relationship confidence between images. The extensive experiments conducted on the publicly available datasets as well as our own challenging dataset demonstrate that the proposed approach can accurately identify image copies from similar images with good interpretability and achieve state-of-the-art performance in detection accuracy and training efficiency.

Research on Classification Method of 3D Point Cloud Model Based on KE-PointVNet

With the popularization and use of three-dimensional data acquisition equipment, the acquisition of three-dimensional data is more convenient. Three-dimensional data contains rich shape and scale information. How to effectively and accurately classify, segment and identify three-dimensional data is a research hotspot in the field of computer vision. Aiming at the particularity of 3D point cloud data and the neglect of local correlation between points, this paper proposes a new end-to-end depth network framework, namely KE-PointVNet. KE-PointVNet can directly deal with point cloud and construct deep convolutional network. KE-PointVNet extracts local geometric features of each point based on the EdgeConv module. By using the KNN proximity algorithm to construct local neighborhood of the point cloud, it can avoid its non-local diffusion, and finally obtain the high-level semantics of the point cloud through the local aggregation vector VLAD layer. The experimental results show that this method has a higher classification accuracy than most of the existing point cloud classification methods.