Semantic Augmentation Research Articles

3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques

Semantic augmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionised image segmentation and classification, its impact on point cloud is an active research field. In this paper, we propose an instance segmentation and augmentation of 3D point clouds using deep learning architectures. We show the potential of an indirect approach using 2D images and a Mask R-CNN (Region-Based Convolution Neural Network). Our method consists of four core steps. We first project the point cloud onto panoramic 2D images using three types of projections: spherical, cylindrical, and cubic. Next, we homogenise the resulting images to correct the artefacts and the empty pixels to be comparable to images available in common training libraries. These images are then used as input to the Mask R-CNN neural network, designed for 2D instance segmentation. Finally, the obtained predictions are reprojected to the point cloud to obtain the segmentation results. We link the results to a context-aware neural network to augment the semantics. Several tests were performed on different datasets to test the adequacy of the method and its potential for generalisation. The developed algorithm uses only the attributes X, Y, Z, and a projection centre (virtual camera) position as inputs.

Investigating Semantic Augmentation in Virtual Environments for Image Segmentation Using Convolutional Neural Networks.

Collecting real-world data for the training of neural networks is enormously time-consuming and expensive. As such, the concept of virtualizing the domain and creating synthetic data has been analyzed in many instances. This virtualization offers many possibilities of changing the domain, and with that, enabling the relatively fast creation of data. It also offers the chance to enhance necessary augmentations with additional semantic information when compared with conventional augmentation methods. This raises the question of whether such semantic changes, which can be seen as augmentations of the virtual domain, contribute to better results for neural networks, when trained with data augmented this way. In this paper, a virtual dataset is presented, including semantic augmentations and automatically generated annotations, as well as a comparison between semantic and conventional augmentation for image data. It is determined that the results differ only marginally for neural network models trained with the two augmentation approaches.

Advances and Challenges for Model and Data Engineering

Advances and Challenges for Model and Data Engineering

DecAug: Out-of-Distribution Generalization via Decomposed Feature Representation and Semantic Augmentation

While deep learning demonstrates its strong ability to handle independent and identically distributed (IID) data, it often suffers from out-of-distribution (OoD) generalization, where the test data come from another distribution (w.r.t. the training one). Designing a general OoD generalization framework for a wide range of applications is challenging, mainly due to different kinds of distribution shifts in the real world, such as the shift across domains or the extrapolation of correlation. Most of the previous approaches can only solve one specific distribution shift, leading to unsatisfactory performance when applied to various OoD benchmarks. In this work, we propose DecAug, a novel decomposed feature representation and semantic augmentation approach for OoD generalization. Specifically, DecAug disentangles the category-related and context-related features by orthogonalizing the two gradients (w.r.t. intermediate features) of losses for predicting category and context labels, where category-related features contain causal information of the target object, while context-related features cause distribution shifts between training and test data. Furthermore, we perform gradient-based augmentation on context-related features to improve the robustness of learned representations. Experimental results show that DecAug outperforms other state-of-the-art methods on various OoD datasets, which is among the very few methods that can deal with different types of OoD generalization challenges.

Semantic segmentation sample augmentation based on simulated scene generation—case study on dock extraction from high spatial resolution imagery

ABSTRACT Deep learning-based semantic segmentation methods, such as fully convolutional networks (FCNs), are state-of-the-art techniques for object extraction from high spatial resolution images. However, collecting massive scene-formed training samples typically required in FCNs is time-consuming and labour-intensive. A suit of automatic sample augmentation schemes based on simulated scene generation is proposed in this study to reduce the manual workload. Proposed schemes include style transfer, target embedding, and mixed modes by utilizing techniques, such as texture transfer, image inpainting, and region-line primitive association framework, which automatically expand the sample set on the basis of a small number of real samples. Dock extraction experiments using UNet are conducted on China’s GaoFen-2 imagery with expanded sample sets. Results showed that the proposed schemes can successfully generate sufficient simulation samples, increase sample diversity, and subsequently improve semantic segmentation accuracy. Compared with the results that use the original real sample set, measures of F 1-score (F 1) and intersection over union (IoU) of dock extraction accuracy demonstrate a maximum improvement of 20.53% and 23.01%, respectively, after sample augmentation.

Generalizable cone beam CT esophagus segmentation using physics-based data augmentation

Automated segmentation of the esophagus is critical in image-guided/adaptive radiotherapy of lung cancer to minimize radiation-induced toxicities such as acute esophagitis. We have developed a semantic physics-based data augmentation method for segmenting the esophagus in both planning CT (pCT) and cone beam CT (CBCT) using 3D convolutional neural networks. One hundred and ninety-one cases with their pCTs and CBCTs from four independent datasets were used to train a modified 3D U-Net architecture and a multi-objective loss function specifically designed for soft-tissue organs such as the esophagus. Scatter artifacts and noises were extracted from week-1 CBCTs using a power-law adaptive histogram equalization method and induced to the corresponding pCT were reconstructed using CBCT reconstruction parameters. Moreover, we leveraged physics-based artifact induction in pCTs to drive the esophagus segmentation in real weekly CBCTs. Segmentations were evaluated using the geometric Dice coefficient and Hausdorff distance as well as dosimetrically using mean esophagus dose and D 5cc. Due to the physics-based data augmentation, our model trained just on the synthetic CBCTs was robust and generalizable enough to also produce state-of-the-art results on the pCTs and CBCTs, achieving Dice overlaps of 0.81 and 0.74, respectively. It is concluded that our physics-based data augmentation spans the realistic noise/artifact spectrum across patient CBCT/pCT data and can generalize well across modalities, eventually improving the accuracy of treatment setup and response analysis.

SA-U-Net++: SAR marine floating raft aquaculture identification based on semantic segmentation and ISAR augmentation

Marine floating raft aquaculture (FRA) monitoring is vital for environment protection and mariculture management. Synthetic aperture radar (SAR) could provide high-quality remote sensing images under all weather conditions compared with the existing optical remote-sensing-based methods. Traditional SAR monitoring methods extract the pixel feature of marine FRA in single patches, which commonly leads to poor generalization. We propose a self-attention semantic segmentation method based on modified U-Net++ (SA-U-Net++) for FRA segmentation, which could automatically extract semantic feature information, and provide superior performance under complicated scenes. The proposed self-attention backbone could help to extract more precise features and enhance the overall accuracy. Furthermore, we propose a FRA-ISAR data generation method based on inverse SAR (ISAR) imaging to alleviate the sample shortage problem. We introduce the semantic segmentation method into FRA-SAR segmentation for the first time. The experiments verify the effectiveness and superiority of SAR FRA segmentation based on the proposed SA-U-Net++ model compared with the existed semantic segmentation approaches.

Regularizing Deep Networks With Semantic Data Augmentation.

Data augmentation is widely known as a simple yet surprisingly effective technique for regularizing deep networks. Conventional data augmentation schemes, e.g., flipping, translation or rotation, are low-level, data-independent and class-agnostic operations, leading to limited diversity for augmented samples. To this end, we propose a novel semantic data augmentation algorithm to complement traditional approaches. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features, i.e., certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., changing the background or view angle of an object. Based on this observation, translating training samples along many such directions in the feature space can effectively augment the dataset for more diversity. To implement this idea, we first introduce a sampling based method to obtain semantically meaningful directions efficiently. Then, an upper bound of the expected cross-entropy (CE) loss on the augmented training set is derived by assuming the number of augmented samples goes to infinity, yielding a highly efficient algorithm. In fact, we show that the proposed implicit semantic data augmentation (ISDA) algorithm amounts to minimizing a novel robust CE loss, which adds minimal extra computational cost to a normal training procedure. In addition to supervised learning, ISDA can be applied to semi-supervised learning tasks under the consistency regularization framework, where ISDA amounts to minimizing the upper bound of the expected KL-divergence between the augmented features and the original features. Although being simple, ISDA consistently improves the generalization performance of popular deep models (e.g., ResNets and DenseNets) on a variety of datasets, i.e., CIFAR-10, CIFAR-100, SVHN, ImageNet, and Cityscapes. Code for reproducing our results is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.

Bearing Fault Diagnosis under Variable Working Conditions Based on Deep Residual Shrinkage Networks and Transfer Learning

Nowadays, deep learning has made great achievements in the field of rotating machinery fault diagnosis. But in the practical engineering scenarios, when facing a large number of unlabeled data and variable operating conditions, only using a deep learning algorithm may reduce the performance. In order to solve the above problem, this paper uses a method of combining transfer learning with deep learning. First, the deep shrinkage residual network is constructed by adding soft thresholds to extract the characteristics of bearing vibration data under noise redundancy. Then, the joint maximum mean deviation (JMMD) criterion and conditional domain adversarial (CDA) learning domain adapting network are used to align the source and target domains. At the same time, adding transferable semantic augmentation (TSA) regular items improves alignment performance between classes. Finally, the proposed model is verified by three experiments: variable load, variable speed, and variable noise, which overcomes the shortcomings of traditional deep learning and shallow transfer learning algorithms.

An augmented representation method of debris flow scenes to improve public perception

ABSTRACT Virtual scenes can present rich and clear disaster information, which can significantly improve the level of public disaster perception. However, existing methods for constructing scenes of debris flow disasters have some deficiencies. First, the construction process does not consider public knowledge, which makes it difficult for the constructed scenes to meet the requirements of the public. Second, the scene representation emphasizes visual effects but lacks augmented visualization, leading to scarcity of semantic information and inefficient public perception. In this paper, the optimal selection of scene objects, semantic augmentation through the combination of various visual variables and dynamic augmented representation are discussed in detail. Finally, a debris flow that occurred Shuimo town is selected for experiment analysis. The experimental results show that most people are unaware of the risks posed by debris flow disasters. The public is more concerned about the consequences of a disaster than its spatiotemporal process, especially when the consequences are related to their own interests. Furthermore, an augmented representation can increase the amount of semantic information of scene objects, which is essential for enhancing public understanding of the causes, processes and effects of debris flows and thereby changing people’s attitudes and enhancing their risk perception.

A Context-Based Disambiguation Model for Sentiment Concepts Using a Bag-of-Concepts Approach

With the widespread dissemination of user-generated content on different social networks, and online consumer systems such as Amazon, the quantity of opinionated information available on the Internet has been increased. One of the main tasks of the sentiment analysis is to detect polarity within a text. The existing polarity detection methods mainly focus on keywords and their naive frequency counts; however, they less regard the meanings and implicit dimensions of the natural concepts. Although background knowledge plays a critical role in determining the polarity of concepts, it has been disregarded in polarity detection methods. This study presents a context-based model to solve ambiguous polarity concepts using commonsense knowledge. First, a model is presented to generate a source of ambiguous sentiment concepts based on SenticNet by computing the probability distribution. Then the model uses a bag-of-concepts approach to remove ambiguities and semantic augmentation with the ConceptNet handling to overcome lost knowledge. ConceptNet is a large-scale semantic network with a large number of commonsense concepts. In this paper, the point mutual information (PMI) measure is used to select the contextual concepts having strong relationships with ambiguous concepts. The polarity of the ambiguous concepts is precisely detected using positive/negative contextual concepts and the relationship of the concepts in the semantic knowledge base. The text representation scheme is semantically enriched using Numberbatch, which is a word embedding model based on the concepts from the ConceptNet semantic network. The proposed model is evaluated by applying a corpus of product reviews, called Semeval. The experimental results revealed an accuracy rate of 82.07%, representing the effectiveness of the proposed model.

Evaluation of standard and semantically-augmented distance metrics for neurology patients

BackgroundPatient distances can be calculated based on signs and symptoms derived from an ontological hierarchy. There is controversy as to whether patient distance metrics that consider the semantic similarity between concepts can outperform standard patient distance metrics that are agnostic to concept similarity. The choice of distance metric can dominate the performance of classification or clustering algorithms. Our objective was to determine if semantically augmented distance metrics would outperform standard metrics on machine learning tasks.MethodsWe converted the neurological findings from 382 published neurology cases into sets of concepts with corresponding machine-readable codes. We calculated patient distances by four different metrics (cosine distance, a semantically augmented cosine distance, Jaccard distance, and a semantically augmented bipartite distance). Semantic augmentation for two of the metrics depended on concept similarities from a hierarchical neuro-ontology. For machine learning algorithms, we used the patient diagnosis as the ground truth label and patient findings as machine learning features. We assessed classification accuracy for four classifiers and cluster quality for two clustering algorithms for each of the distance metrics.ResultsInter-patient distances were smaller when the distance metric was semantically augmented. Classification accuracy and cluster quality were not significantly different by distance metric.ConclusionAlthough semantic augmentation reduced inter-patient distances, we did not find improved classification accuracy or improved cluster quality with semantically augmented patient distance metrics when applied to a dataset of neurology patients. Further work is needed to assess the utility of semantically augmented patient distances.

A novel strategy to balance the results of cross-modal hashing

Hashing methods for cross-modal retrieval has drawn increasing research interests and has been widely studied in recent years due to the explosive growth of multimedia big data. However, a significant phenomenon which has been ignored is that there is a large gap between the results of cross-modal hashing in most cases. For example, the results of Text-to-Image frequently outperform that of Image-to-Text with a large margin. In this paper, we propose a strategy named semantic augmentation to improve and balance the results of cross-modal hashing. An intermediate semantic space is constructed to re-align the feature representations that embedded with weak semantic information. By using the intermediate semantic space, the semantic information of visual features can be further augmented before being sent to cross-modal hashing algorithms. Extensive experiments are carried out on four datasets via seven state-of-the-art cross-modal hashing methods. Compared against the results without semantic augmentation, the Image-to-Text results of these methods with semantic augmentation are improved considerably, which demonstrates the effectiveness of the proposed semantic augmentation strategy in bridging the gap between the results of cross-modal retrieval. Additional experiments are conducted on the real-valued, semi-supervised, semi-paired, partial-paired, and unpaired cross-modal retrieval methods, the results further indicates the effectiveness of our strategy in improving performance of cross-modal retrieval.

Text classification with semantically enriched word embeddings

AbstractThe recent breakthroughs in deep neural architectures across multiple machine learning fields have led to the widespread use of deep neural models. These learners are often applied as black-box models that ignore or insufficiently utilize a wealth of preexisting semantic information. In this study, we focus on the text classification task, investigating methods for augmenting the input to deep neural networks (DNNs) with semantic information. We extract semantics for the words in the preprocessed text from the WordNet semantic graph, in the form of weighted concept terms that form a semantic frequency vector. Concepts are selected via a variety of semantic disambiguation techniques, including a basic, a part-of-speech-based, and a semantic embedding projection method. Additionally, we consider a weight propagation mechanism that exploits semantic relationships in the concept graph and conveys a spreading activation component. We enrich word2vec embeddings with the resulting semantic vector through concatenation or replacement and apply the semantically augmented word embeddings on the classification task via a DNN. Experimental results over established datasets demonstrate that our approach of semantic augmentation in the input space boosts classification performance significantly, with concatenation offering the best performance. We also note additional interesting findings produced by our approach regarding the behavior of term frequency - inverse document frequency normalization on semantic vectors, along with the radical dimensionality reduction potential with negligible performance loss.

Multi-scale semantic feature fusion and data augmentation for acoustic scene classification

This paper investigates a multi-scale semantic feature fusion and data augmentation approach for deep convolutional neural network (CNN) based acoustic scene classification. To ensemble the multi-scale semantic information of CNN and improve the performance of acoustic scene classification, a multi-scale feature fusion framework, which consists of a simplified Xception backbone and a semantic feature fusion strategy, is presented. A novel label smoothing mixup data augmentation method, which is a generalization of mixup and label smoothing, is proposed to alleviate the over-confident problem of network training. A spatial-mixup technique is presented to generate meaningful mixup virtual data for acoustic scene classification. Extensive experiments on synthetic data and real acoustic scene classification dataset demonstrate that both multi-scale semantic feature fusion and label smoothing spatial-mixup data augmentation are effective for improving the acoustic scene classification performance of a deep neural network.

Entity Thematic Similarity Measurement for Personal Explainable Searching Services in the Edge Environment

Currently, search engines are widely used to address the information overload problem. Different from the existing client-and-sever-based frameworks, edge computing (EC) technology can provide a new architecture for personalized searching services. The issue of how to measure the similarities among entities by using the context information generated by user behavior in the edge environment is vital in the task of entity-related personal searching. To analyze and measure the similarities among entities, existing methods are mainly based on either the textual content or relationships unilaterally, and the results usually have a fixed degree of similarity. However, the similarities among entities depend on the set of properties that belong to the entities. This approach should be used in determining the similarity or dissimilarity associated with the surrounding context. To address this limitation, we propose a novel semantic augmentation method with a double attention mechanism. The method refers to a dynamic representation learning process that maps an entity to a real number vector in semantic space. In this article, different from the existing similarity measurement methods, we propose a thematic similarity measure approach to analyze the connotation and denotation similarities among entities. The experimental results show that the double attention mechanism leads to a significant improvement in the entity thematic similarity measurement tasks. The model can make a separation among the entities from different domains effectively. In addition, it can take similar entities that are closer in the same domain. It also shows excellent performance on the task of entity thematic similarity, which makes the recommendation results more explainable.

SEMANTIC ENRICHMENT OF POINT CLOUD BY AUTOMATIC EXTRACTION AND ENHANCEMENT OF 360° PANORAMAS

Abstract. The raw nature of point clouds is an important challenge for their direct exploitation in architecture, engineering and construction applications. Particularly, their lack of semantics hinders their utility for automatic workflows (Poux, 2019). In addition, the volume and the irregularity of the structure of point clouds makes it difficult to directly and automatically classify datasets efficiently, especially when compared to the state-of-the art 2D raster classification. Recently, with the advances in deep learning models such as convolutional neural networks (CNNs) , the performance of image-based classification of remote sensing scenes has improved considerably (Chen et al., 2018; Cheng et al., 2017). In this research, we examine a simple and innovative approach that represent large 3D point clouds through multiple 2D projections to leverage learning approaches based on 2D images. In other words, the approach in this study proposes an automatic process for extracting 360° panoramas, enhancing these to be able to leverage raster data to obtain domain-base semantic enrichment possibilities. Indeed, it is very important to obtain a rigorous characterization for use in the classification of a point cloud. Especially because there is a very large variety of 3D point cloud domain applications. In order to test the adequacy of the method and its potential for generalization, several tests were performed on different datasets. The developed semantic augmentation algorithm uses only the attributes X, Y, Z and camera positions as inputs.

A hybrid structured deep neural network with Word2Vec for construction accident causes classification

According to the latest fatal work injury rates reported by the Bureau of Labors Statistics, construction sites remain the most hazardous workplaces. In the construction sector, fatality investigation summary reports are available for past accidents and by investigating such reports, valuable insights can be gained. In this study, text mining algorithms are explored for automatic construction accident causes classification. To be more specific, Word2Vec skip-gram model is utilized to learn word embedding from a domain-specific corpus and a hybrid structured deep neural network is proposed by incorporating the learned word embedding for accident reports classification. Dataset from Occupational Safety and Health Administration (OSHA) is employed in the experiment to evaluate the performance of the proposed approach. Besides, five baseline models: support vector machine (SVM), linear regression (LR), K-nearest neighbor (KNN), decision tree (DT), Naive Bayes (NB) are employed to compare with the proposed approach. Experiment results show that the proposed model achieves the highest average weighted F1 score among all models considered in this study. The result also proves the effectiveness of applying Word2Vec skip-gram algorithm for semantic information augmentation. As a result, robustness of the model is improved when classifying cases of low support values.

Semantic three-stream network for social relation recognition

In this paper, we propose a semantic three-stream network (STN) for social relation recognition, which learns discriminative features from facial images directly by exploiting semantic information effectively. Specifically, we employ a semantic augmentation structure to extract enriched semantic features from original face images, where a Siamese network is used to extract features from a pair of face images. We concatenate features of three streams for social relation recognition. Unlike most existing relation recognition methods, our method explicitly uses semantic information to discover the social relation. The proposed semantic augmentation structure can be easily embedded into the off-the-shell deep neural network, which leads to a powerful and flexible semantic augmentation network. Experimental results show that our proposed method outperforms the state-of-the-arts.

A Machine Learning based Preventing the Occurrence of Cyber Bullying Messages on OSN

The process of threaten or harassment of any user with the help of posting wrong/abused or vulgar messages using the social media in the internet is known as Cyber bullying .These messages may sometime contain a text posted by a teen, or preteen or a child who want to threaten or harassed or embarrassed other child by posting the messages. So in this project, we mainly try to propose another depiction learning strategy to handle this issue known as SEMdae. Here the semantic augmentation comprises of predefined words that contain noise or abused meaning which is posted into the database by the admin and these words are classified based on the five categories that are available in the literature like “HATE, VULGAR, OFFENSIVE, SEX, and VOILENCE”.