High-level Semantics Research Articles

Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis

Existing physical cloth simulators suffer from expensive computation and difficulties in tuning mechanical parameters to get desired wrinkling behaviors. Data-driven methods provide an alternative solution. It typically synthesizes cloth animation at a much lower computational cost, and also creates wrinkling effects that highly resemble the much controllable training data. In this paper we propose a deep learning based method for synthesizing cloth animation with high resolution meshes. To do this we first create a dataset for training: a pair of low and high resolution meshes are simulated and their motions are synchronized. As a result the two meshes exhibit similar large-scale deformation but different small wrinkles. Each simulated mesh pair are then converted into a pair of low and high resolution "images" (a 2D array of samples), with each sample can be interpreted as any of three features: the displacement, the normal and the velocity. With these image pairs, we design a multi-feature super-resolution (MFSR) network that jointly train an upsampling synthesizer for the three features. The MFSR architecture consists of two key components: a sharing module that takes multiple features as input to learn low-level representations from corresponding super-resolution tasks simultaneously; and task-specific modules focusing on various high-level semantics. Frame-to-frame consistency is well maintained thanks to the proposed kinematics-based loss function. Our method achieves realistic results at high frame rates: 12-14 times faster than traditional physical simulation. We demonstrate the performance of our method with various experimental scenes, including a dressed character with sophisticated collisions.

FE-RetinaNet: Small Target Detection with Parallel Multi-Scale Feature Enhancement

Because small targets have fewer pixels and carry fewer features, most target detection algorithms cannot effectively use the edge information and semantic information of small targets in the feature map, resulting in low detection accuracy, missed detections, and false detections from time to time. To solve the shortcoming of insufficient information features of small targets in the RetinaNet, this work introduces a parallel-assisted multi-scale feature enhancement module MFEM (Multi-scale Feature Enhancement Model), which uses dilated convolution with different expansion rates to avoid multiple down sampling. MFEM avoids information loss caused by multiple down sampling, and at the same time helps to assist shallow extraction of multi-scale context information. Additionally, this work adopts a backbone network improvement plan specifically designed for target detection tasks, which can effectively save small target information in high-level feature maps. The traditional top-down pyramid structure focuses on transferring high-level semantics from the top to the bottom, and the one-way information flow is not conducive to the detection of small targets. In this work, the auxiliary MFEM branch is combined with RetinaNet to construct a model with a bidirectional feature pyramid network, which can effectively integrate the strong semantic information of the high-level network and high-resolution information regarding the low level. The bidirectional feature pyramid network designed in this work is a symmetrical structure, including a top-down branch and a bottom-up branch, performs the transfer and fusion of strong semantic information and strong resolution information. To prove the effectiveness of the algorithm FE-RetinaNet (Feature Enhancement RetinaNet), this work conducts experiments on the MS COCO. Compared with the original RetinaNet, the improved RetinaNet has achieved a 1.8% improvement in the detection accuracy (mAP) on the MS COCO, and the COCO AP is 36.2%; FE-RetinaNet has a good detection effect on small targets, with APs increased by 3.2%.

Disentangled Representation Learning in Heterogeneous Information Network for Large-scale Android Malware Detection in the COVID-19 Era and Beyond

In the fight against the COVID-19 pandemic, many social activities have moved online; society's overwhelming reliance on the complex cyberspace makes its security more important than ever. In this paper, we propose and develop an intelligent system named Dr.HIN to protect users against the evolving Android malware attacks in the COVID-19 era and beyond. In Dr.HIN, besides app content, we propose to consider higher-level semantics and social relations among apps, developers and mobile devices to comprehensively depict Android apps; and then we introduce a structured heterogeneous information network (HIN) to model the complex relations and exploit meta-path guided strategy to learn node (i.e., app) representations from HIN. As the representations of malware could be highly entangled with benign apps in the complex ecosystem of development, it poses a new challenge of learning the latent explanatory factors hidden in the HIN embeddings to detect the evolving malware. To address this challenge, we propose to integrate domain priors generated from different views (i.e., app content, app authorship, app installation) to devise an adversarial disentangler to separate the distinct, informative factors of variations hidden in the HIN embeddings for large-scale Android malware detection. This is the first attempt of disentangled representation learning in HIN data. Promising experimental results based on the large-scale and real sample collections from security industry demonstrate the performance of Dr.HIN in evolving Android malware detection, by comparison with baselines and popular mobile security products.

Multi-Level and Multi-Scale Feature Aggregation Network for Semantic Segmentation in Vehicle-Mounted Scenes.

The main challenges of semantic segmentation in vehicle-mounted scenes are object scale variation and trading off model accuracy and efficiency. Lightweight backbone networks for semantic segmentation usually extract single-scale features layer-by-layer only by using a fixed receptive field. Most modern real-time semantic segmentation networks heavily compromise spatial details when encoding semantics, and sacrifice accuracy for speed. Many improving strategies adopt dilated convolution and add a sub-network, in which either intensive computation or redundant parameters are brought. We propose a multi-level and multi-scale feature aggregation network (MMFANet). A spatial pyramid module is designed by cascading dilated convolutions with different receptive fields to extract multi-scale features layer-by-layer. Subseqently, a lightweight backbone network is built by reducing the feature channel capacity of the module. To improve the accuracy of our network, we design two additional modules to separately capture spatial details and high-level semantics from the backbone network without significantly increasing the computation cost. Comprehensive experimental results show that our model achieves 79.3% MIoU on the Cityscapes test dataset at a speed of 58.5 FPS, and it is more accurate than SwiftNet (75.5% MIoU). Furthermore, the number of parameters of our model is at least 53.38% less than that of other state-of-the-art models.

Self-attention feature fusion network for semantic segmentation

Capturing rich contextual information is very helpful for semantic segmentation. In addition, the effective use of low-level details and high-level semantics is crucial for semantic segmentation. In this paper, we start from these two aspects, and we propose a self-attention feature fusion network for semantic segmentation (SA-FFNet) to improve semantic segmentation performance. Specifically, we introduced the vertical and horizontal compression attention module (VH-CAM) and the unequal channel pyramid pooling module (UC-PPM). The previous position attention module, such as the position attention module (PAM) in DANet, calculates the similarity between each pixel and all other pixels. However, the amount of information contained in a single isolated pixel is too small, so the resulting position attention weight is not perfect. Our approach is to compress a feature map from both vertical and horizontal directions so that the information contained in each pixel will be richer, and the spatial feature map obtained from this will be better. Experiments show that the position attention weight generated by our vertical and horizontal compression attention module is better than PAM in DANet. Additionally, our UC-PPM on each decoder can provide high-level rich semantic information to guide the selection of low-level feature maps. The proposed model achieves a 76.42% mean IoU on PASCAL VOC2012 and a 73.13% mean IoU on Cityscapes.

Dual-Path Sparse Hierarchical Network for Semantic Segmentation of Remote Sensing Images

Semantic segmentation of remote sensing images aims to label every pixel with the correct semantic category. The core challenge of the current deep convolutional network (ConvNet)-based methods lies in the difficulty of effectively aggregating high-level categorical semantics and low-level local details along the hierarchy of backbone. Most current approaches consider only fusing adjacent feature layers gradually with short-range feature connections, which lack the diversity of feature interactions, such as long-range cross-scale connections. To this end, we propose a novel dual-path sparse hierarchical network that is characterized by rich cross-scale feature interactions. Multiscale features are first sparsely grouped with a predefined interval, which is then aggregated via both long-range and short-range cross-scale connections in a hierarchical manner. Moreover, in order to further enrich the diversity of feature interactions, we also introduce another fusion path in parallel but with different sparsity for feature grouping, forming a dual-path network. In this way, our model is able to effectively aggregate multilevel features by incorporating both long-range and short-range feature interactions in both parallel and hierarchical manner. Meanwhile, the semantic and resolution gap between multilevel features can also be bridged.

Semi-supervised semantic segmentation in Earth Observation: the MiniFrance suite, dataset analysis and multi-task network study

The development of semi-supervised learning techniques is essential to enhance the generalization capacities of machine learning algorithms. Indeed, raw image data are abundant while labels are scarce, therefore it is crucial to leverage unlabeled inputs to build better models. The availability of large databases have been key for the development of learning algorithms with high level performance. Despite the major role of machine learning in Earth Observation to derive products such as land cover maps, datasets in the field are still limited, either because of modest surface coverage, lack of variety of scenes or restricted classes to identify. We introduce a novel large-scale dataset for semi-supervised semantic segmentation in Earth Observation, the MiniFrance suite. MiniFrance has several unprecedented properties: it is large-scale, containing over 2000 very high resolution aerial images, accounting for more than 200 billions samples (pixels); it is varied, covering 16 conurbations in France, with various climates, different landscapes, and urban as well as countryside scenes; and it is challenging, considering land use classes with high-level semantics. Nevertheless, the most distinctive quality of MiniFrance is being the only dataset in the field especially designed for semi-supervised learning: it contains labeled and unlabeled images in its training partition, which reproduces a life-like scenario. Along with this dataset, we present tools for data representativeness analysis in terms of appearance similarity and a thorough study of MiniFrance data, demonstrating that it is suitable for learning and generalizes well in a semi-supervised setting. Finally, we present semi-supervised deep architectures based on multi-task learning and the first experiments on MiniFrance. These results will serve as baselines for future work on semi-supervised learning over the MiniFrance dataset. The Minifrance suite and related semi-supervised networks will be publicly available to promote semi-supervised works in Earth Observation.

Edge-aware salient object detection network via context guidance

Fully convolutional network (FCN) based salient object detection methods have shown their advantages in highlighting salient regions because they can obtain global semantic information. And the high-level semantics are usually passed in a top-down pathway. However, the semantic information would be diluted progressively among different level features. To alleviate this issue, we propose a novel edge-aware salient object detection network. Our network utilizes high-level semantic information to assist the feature selection of shallower layers. Specifically, we extract refined features from different levels of the backbone. Then, we obtain global contextual information to locate the salient objects by extracting multi-scale features and emphasizing the important feature channels. In order to assist the shallower layers to pay attention to the learning of meaningful local information, we adopt a context guidance strategy to fuse the high-level and low-level information. Finally, we supervise the generation of low-level edge information to preserve the salient object boundaries. Extensive experiments demonstrate that the proposed mode performs favorably against most state-of-the-art methods under different evaluation metrics on six popular benchmarks.

Consolidated domain adaptive detection and localization framework for cross-device colonoscopic images

Automatic polyp detection has been proven to be crucial in improving the diagnosis accuracy and reducing colorectal cancer mortality during the precancerous stage. However, the performance of deep neural networks may degrade severely when being deployed to polyp data in a distinct domain. This domain distinction can be caused by different scanners, hospitals, or imaging protocols. In this paper, we propose a consolidated domain adaptive detection and localization framework to bridge the domain gap between different colonosopic datasets effectively, consisting of two parts: the pixel-level adaptation and the hierarchical feature-level adaptation. For the pixel-level adaptation part, we propose a Gaussian Fourier Domain Adaptation (GFDA) method to sample the matched source and target image pairs from Gaussian distributions then unify their styles via the low-level spectrum replacement, which can reduce the domain discrepancy of the cross-device polyp datasets in appearance level without distorting their contents. The hierarchical feature-level adaptation part comprising a Hierarchical Attentive Adaptation (HAA) module to minimize the domain discrepancy in high-level semantics and an Iconic Concentrative Adaptation (ICA) module to perform reliable instance alignment. These two modules are regularized by a Generalized Consistency Regularizer (GCR) for maintaining the consistency of their domain predictions. We further extend our framework to the polyp localization task and present a Centre Besiegement (CB) loss for better location optimization. Experimental results show that our framework outperforms other domain adaptation detectors by a large margin in the detection task meanwhile achieves the state-of-the-art recall rate of 87.5% in the localization task. The source code is available at https://github.com/CityU-AIM-Group/ConsolidatedPolypDA.

Single annotated pixel based weakly supervised semantic segmentation under driving scenes

Semantic segmentation tasks based on weakly supervised conditions have been put forward to achieve a lightweight labeling process. For simple images that only include a few categories, research based on image-level annotations has achieved acceptable performance. However, when facing complex scenes, since image contains a large number of classes, it becomes challenging to learn visual appearance based on image tags. In this case, image-level annotations are not useful in providing information. Therefore, we set up a new task in which a single annotated pixel is provided for each category in a whole dataset. Based on the more lightweight and informative condition, a three step process is built for pseudo labels generation, which progressively implements each class’ optimal feature representation, image inference, and context-location based refinement. In particular, since high-level semantics and low-level imaging features have different discriminative abilities for each class under driving scenes, we divide categories into “object” or “scene” and then provide different operations for the two types separately. Further, an alternate iterative structure is established to gradually improve segmentation performance, which combines CNN-based inter-image common semantic learning and imaging prior based intra-image modification process. Experiments on the Cityscapes dataset demonstrate that the proposed method provides a feasible way to solve weakly supervised semantic segmentation tasks under complex driving scenes.

Scene change detection: semantic and depth information

Change detection is widely used in city construction, remote image detection, autonomous driving and so on. The most important challenge in change detection is how to define the real change, that is to say, how to detect the real changes while ignoring the unexpected changes? such as varying illumination, shadows, seasonal scene changes. Motivated by the key issue, in this paper, we propose a novel two-classified method that utilizes the depth information to assist the semantic information for better detecting changes. A gradual modification strategy is also designed to combine the high-level semantics and the low-level edge-sensitive features to achieve better depth estimation. The proposed framework was evaluated by using the VL-CMU-CD streetscape change detection dataset. Both quantitative and qualitative experiments have been implemented for evaluating the performance of the framework under different light and seasons. Experimental results show that the proposed method outperforms most of the current state-of-the-art results.

Learning Semantic Features for Classifying Very Large Image Datasets Using Convolution Neural Network

Advancements in sensors and image acquisition devices lead to tremendous increase in creation of unlabeled database of images, and traditional image retrieval approaches are inefficient in retrieving semantic images. Neural network is emerging as popular method to solve most of the state-of-the-art problems in filling the gap between low level features and high level semantics. Convolution neural network (CNN) is a category of neural network which automatically extracts the important features without any human intervention, with considerably reduced set of parameters. In this paper, a model is generated using CNN (VGG-16) architecture which combines convolution and max pooling layers at different levels using effective regularization and transfer learning with data augmentation. The effectiveness of proposed model is demonstrated on benchmarking Caltech 101 and 256 datasets. The results obtained have proven the capacity of model in understanding high level semantics and outperformed several contemporary techniques.

Knowledge and Spatial Pyramid Distance-Based Gated Graph Attention Network for Remote Sensing Semantic Segmentation

The pixel-based semantic segmentation methods take pixels as recognitions units, and are restricted by the limited range of receptive fields, so they cannot carry richer and higher-level semantics. These reduce the accuracy of remote sensing (RS) semantic segmentation to a certain extent. Comparing with the pixel-based methods, the graph neural networks (GNNs) usually use objects as input nodes, so they not only have relatively small computational complexity, but also can carry richer semantic information. However, the traditional GNNs are more rely on the context information of the individual samples and lack geographic prior knowledge that reflects the overall situation of the research area. Therefore, these methods may be disturbed by the confusion of “different objects with the same spectrum” or “violating the first law of geography” in some areas. To address the above problems, we propose a remote sensing semantic segmentation model called knowledge and spatial pyramid distance-based gated graph attention network (KSPGAT), which is based on prior knowledge, spatial pyramid distance and a graph attention network (GAT) with gating mechanism. The model first uses superpixels (geographical objects) to form the nodes of a graph neural network and then uses a novel spatial pyramid distance recognition algorithm to recognize the spatial relationships. Finally, based on the integration of feature similarity and the spatial relationships of geographic objects, a multi-source attention mechanism and gating mechanism are designed to control the process of node aggregation, as a result, the high-level semantics, spatial relationships and prior knowledge can be introduced into a remote sensing semantic segmentation network. The experimental results show that our model improves the overall accuracy by 4.43% compared with the U-Net Network, and 3.80% compared with the baseline GAT network.

Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder.

While vision evokes a dense network of feedforward and feedback neural processes in the brain, visual processes are primarily modeled with feedforward hierarchical neural networks, leaving the computational role of feedback processes poorly understood. Here, we developed a generative autoencoder neural network model and adversarially trained it on a categorically diverse data set of images. We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. We compared representational similarity of the activity patterns in the proposed model with temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) visual brain responses. The proposed generative model identified two segregated neural dynamics in the visual brain. A temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally later dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Our results append to previous studies on neural feedback processes by presenting a new insight into the algorithmic function and the information carried by the feedback processes in the ventral visual pathway.

Semantic-enhanced sequential modeling for personality trait recognition from texts

The automatic recognition of personality traits from texts has attracted significant attention. Existing studies typically combine linguistic feature engineering with traditional models, use five various neural networks to predict personality traits with multiple labels, and fail to achieve the best performance on each label. To this end, in this paper, we propose a novel semantic-enhanced personality recognition neural network (SEPRNN) model, which has a goal of avoiding dependence on feature engineering, allowing the same model to adapt to detecting five various personality traits with no modification to the model itself, and employing deep learning based methods and atomic features of texts to build vectorial word-level representation for personality trait recognition. Specifically, to precisely recognize multi-labeled personality traits, we first propose a word-level semantic representation for texts based on context learning. Then, a fully connected layer is used to obtain higher-level semantics of texts. Finally, the experimental results demonstrate that the proposed approach achieves significant performance improvement for multi-labeled personality traits compared with several baselines.

RETRACTED ARTICLE: Research on traditional art digitization based on machine learning and image recognition

The stage art of Peking Opera is very charming and exudes dazzling light in many aspects. The facial makeup art has to be mentioned. In recent years, its value in aesthetics and application has also increased. In the rapidly changing modern society, people see a lot of digital images every day, and how to research and apply a large number of digital images has also become the focus of many people, and the underlying features such as texture, shape and color can help very well. We distinguish between digital images and at the same time establish connections with high-level semantics. This paper mainly studies the facial makeup images of Peking Opera and their recognition methods. There are two problems. First, compared with database images, images taken with mobile phones and other devices that can change their positions must undergo a series of related processing; second, how to meet the requirements based on recognition efficiency demand for mobile applications.

IMPROVED SYSTEM FOR RETRIEVAL OF COLOR LOGO IMAGES USING PSO, SOM AND RELEVANCE FEEDBACK TECHNIQUE

The colour logo identification has one of the key problems of bridging the difference between low-level characteristics and high-level semantics, as the consumer wants to. Relevance feedback (RF) along with self-organizing map (SOM) have been successfully introduced to resolve this void. However, the efficiency of the automated map (SOM) based RF is always low when the feedback sample is limited. This paper proposed a new technology, namely the SOM-SOM-RF that combines SOM-based RF with warm particle optimization, to boost RF performance (PSO). The aim of this proposed technology is to increase SOM-based RF efficiency and also to minimise user encounters with the device by reducing its RF number. On 3000 colour logo pictures, the PSO-SOM-RF was tested. The findings from the tests revealed the high precision of the proposed PSO-SOM-RF.

Feature Extraction and Retrieval of Ecommerce Product Images Based on Image Processing

The new retail is an industry featured by online ecommerce. One of the key techniques of the industry is the product identification based on image processing. This technique has an important business application value, because it is capable of improving the retrieval efficiency of products and the level of information supervision. To acquire high-level semantics of images and enhance the retrieval effect of products, this paper explores the feature extraction and retrieval of ecommerce product images based on image processing. The improved Fourier descriptor was innovatively into a metric learning-based product image feature extraction network, and the attention mechanism was introduced to realize accurate retrieval of product images. Firstly, the authors detailed how to acquire the product contour and the axis with minimum moment of inertia, and then extracted the shape feature of products. Next, a feature extraction network was established based on the metric learning supervision, which is capable of obtaining distinctive feature, and thus realized the extraction of distinctive and classification features of products. Finally, the authors expounded on the product image retrieval method based on cluster attention neural network. The effectiveness of our method was confirmed through experiments. The research results provide a reference for feature extraction and retrieval in other fields of image processing.

A Weakly Supervised Semantic Segmentation Network by Aggregating Seed Cues: The Multi-Object Proposal Generation Perspective

Weakly supervised semantic segmentation under image-level annotations is effectiveness for real-world applications. The small and sparse discriminative regions obtained from an image classification network that are typically used as the important initial location of semantic segmentation also form the bottleneck. Although deep convolutional neural networks (DCNNs) have exhibited promising performances for single-label image classification tasks, images of the real-world usually contain multiple categories, which is still an open problem. So, the problem of obtaining high-confidence discriminative regions from multi-label classification networks remains unsolved. To solve this problem, this article proposes an innovative three-step framework within the perspective of multi-object proposal generation. First, an image is divided into candidate boxes using the object proposal method. The candidate boxes are sent to a single-classification network to obtain the discriminative regions. Second, the discriminative regions are aggregated to obtain a high-confidence seed map. Third, the seed cues grow on the feature maps of high-level semantics produced by a backbone segmentation network. Experiments are carried out on the PASCAL VOC 2012 dataset to verify the effectiveness of our approach, which is shown to outperform other baseline image segmentation methods.

Learning to rank implicit entities on Twitter

Linking textual content to entities from the knowledge graph has received increasing attention in the context of which surface form representations of entities, e.g., terms or phrases, are disambiguated and linked to appropriate entities. This allows textual content, e.g., social user-generated content, to be interpreted and reasoned on at a higher semantic level. However, recent research has shown that at least 15% of social user-generated content do not have explicit surface form representation of entities that they discuss. In other words, the subject of the content is only implied. For such cases, existing entity linking methods, known as explicit entity linking, cannot perform linking because entity surface form is missing. In this paper, we investigate how implicit entities within social content can be identified and linked. The contributions of our work include (1) modeling the problem of implicit entity linking as a learn to rank problem where knowledge graph entities are ranked based on their relevance to the input tweet, (2) the introduction and systematic classification of appropriate features for identifying implicit entities, (3) extensive evaluation of the proposed approach in comparison with existing state of the art as well as performing feature analysis over proposed features, and (4) the qualitative assessment of the root causes for mislabeled instances in our experiments and careful discussion on how mislabeled entity links can be addressed as a part of future work. In our experiments, we show that our proposed features are able to improve the state of the art over the standard Precision at 1 (P@1) metric.