Semantic Prediction Research Articles

JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds

In this paper, we propose a novel joint instance and semantic segmentation approach, which is called JSNet, in order to address the instance and semantic segmentation of 3D point clouds simultaneously. Firstly, we build an effective backbone network to extract robust features from the raw point clouds. Secondly, to obtain more discriminative features, a point cloud feature fusion module is proposed to fuse the different layer features of the backbone network. Furthermore, a joint instance semantic segmentation module is developed to transform semantic features into instance embedding space, and then the transformed features are further fused with instance features to facilitate instance segmentation. Meanwhile, this module also aggregates instance features into semantic feature space to promote semantic segmentation. Finally, the instance predictions are generated by applying a simple mean-shift clustering on instance embeddings. As a result, we evaluate the proposed JSNet on a large-scale 3D indoor point cloud dataset S3DIS and a part dataset ShapeNet, and compare it with existing approaches. Experimental results demonstrate our approach outperforms the state-of-the-art method in 3D instance segmentation with a significant improvement in 3D semantic prediction and our method is also beneficial for part segmentation. The source code for this work is available at https://github.com/dlinzhao/JSNet.

How top-down processing enhances comprehension of noise-vocoded speech: Predictions about meaning are more important than predictions about form

Abstract Listeners quickly learn to understand speech that has been distorted, and this process is enhanced when comprehension is constrained by higher-level knowledge. In three experiments, we investigated whether this knowledge enhances comprehension of distorted speech because it allows listeners to predict (1) the meaning of the distorted utterance, or (2) the lower-level wordforms. Participants listened to question-answer sequences, in which questions were clearly-spoken but answers were noise-vocoded. Comprehension (Experiment 1) and learning (Experiment 2) were enhanced when listeners could use the question to predict the semantics of the distorted answer, but were not enhanced by predictions of answer form. Form predictions enhanced comprehension only when questions and answers were significantly separated by time and intervening linguistic material (Experiment 3). Together, these results suggest that high-level semantic predictions enhance comprehension and learning, with form predictions playing only a minimal role.

Data mining model based on user reviews and star ratings

With the rapid development of e-commerce, the research on sentiment analysis of online reviews has been paid more and more attention. This paper presents an Aspect-Level Sentiment Analysis Method based on long short-term memory (LSTM) and boot-strapping, which performs semantic mining and prediction on time-based data patterns and data combinations of text, star rating and helpful votes. A high prediction accuracy rate is obtained in the open data set. Compared with the traditional methods, which single analysis comment or evaluation, merchants can gain a deeper understanding of user feedback from sentiment analysis.

A Tale of Two Positivities and the N400: Distinct Neural Signatures Are Evoked by Confirmed and Violated Predictions at Different Levels of Representation.

It has been proposed that hierarchical prediction is a fundamental computational principle underlying neurocognitive processing. Here, we ask whether the brain engages distinct neurocognitive mechanisms in response to inputs that fulfill versus violate strong predictions at different levels of representation during language comprehension. Participants read three-sentence scenarios in which the third sentence constrained for a broad event structure, for example, {Agent caution animate-Patient}. High constraint contexts additionally constrained for a specific event/lexical item, for example, a two-sentence context about a beach, lifeguards, and sharks constrained for the event, {Lifeguards cautioned Swimmers}, and the specific lexical item swimmers. Low constraint contexts did not constrain for any specific event/lexical item. We measured ERPs on critical nouns that fulfilled and/or violated each of these constraints. We found clear, dissociable effects to fulfilled semantic predictions (a reduced N400), to event/lexical prediction violations (an increased late frontal positivity), and to event structure/animacy prediction violations (an increased late posterior positivity/P600). We argue that the late frontal positivity reflects a large change in activity associated with successfully updating the comprehender's current situation model with new unpredicted information. We suggest that the late posterior positivity/P600 is triggered when the comprehender detects a conflict between the input and her model of the communicator and communicative environment. This leads to an initial failure to incorporate the unpredicted input into the situation model, which may be followed by second-pass attempts to make sense of the discourse through reanalysis, repair, or reinterpretation. Together, these findings provide strong evidence that confirmed and violated predictions at different levels of representation manifest as distinct spatiotemporal neural signatures.

Scale-aware spatial pyramid pooling with both encoder-mask and scale-attention for semantic segmentation

This paper focuses on semantic segmentation of scenes by capturing the appropriate scale, rich detail and contextual dependencies in a feature representation. Semantic segmentation is a pixel-level classification task and has made steady progress on the basis of fully convolutional networks (FCNs). However, we find there is still room for improvement in the following aspects. The first is that the pixel itself has not enough information for semantic prediction, it needs to look around to determine which category it belongs to. However, the fixed size of the receptive field defined by the network is not suitable for all pixels when an image contains objects with various scales. The second is that the extracted scale-aware features do not handle sharper object boundaries due to low-resolution. The final aspect is regarding the ability to model long-range dependencies. In order to solve the above challenges, in this paper we propose three modules: Scale-aware spatial pyramid pool module, Encoder mask module and Scale-attention module (SSPP-ES). Extensive experiments on the Cityscapes and ADE20K benchmarks demonstrate the effectiveness of our approach for semantic segmentation.

Building segmentation through a gated graph convolutional neural network with deep structured feature embedding

Automatic building extraction from optical imagery remains a challenge due to, for example, the complexity of building shapes. Semantic segmentation is an efficient approach for this task. The latest development in deep convolutional neural networks (DCNNs) has made accurate pixel-level classification tasks possible. Yet one central issue remains: the precise delineation of boundaries. Deep architectures generally fail to produce fine-grained segmentation with accurate boundaries due to their progressive down-sampling. Hence, we introduce a generic framework to overcome the issue, integrating the graph convolutional network (GCN) and deep structured feature embedding (DSFE) into an end-to-end workflow. Furthermore, instead of using a classic graph convolutional neural network, we propose a gated graph convolutional network, which enables the refinement of weak and coarse semantic predictions to generate sharp borders and fine-grained pixel-level classification. Taking the semantic segmentation of building footprints as a practical example, we compared different feature embedding architectures and graph neural networks. Our proposed framework with the new GCN architecture outperforms state-of-the-art approaches. Although our main task in this work is building footprint extraction, the proposed method can be generally applied to other binary or multi-label segmentation tasks.

Agreement Mismatch in Partitive Relatives

Agreement Mismatch in Partitive Relatives

Effective image restoration for semantic segmentation

Recent semantic segmentation algorithms are greatly accelerated by deep convolutional neural networks (DCNNs). Although most of them perform well on normal images, they are not robust to the degenerations of images. To boost the performance of semantic segmentation on degraded images, we present an effective image restoration framework based on generative adversarial network (GAN). Firstly, we propose to jointly minimize pixel-wise cross entropy loss and our semantic-aware mean square error loss for the optimization of generative restoration network, which ensures correct semantic predictions at each pixel location. Secondly, we introduce GAN into our framework, which can benefit high-order consistency of semantic predictions. Comprehensive experimental results on image super-resolution and denoising demonstrate that our approach is able to achieve the best segmentation accuracy as well as maintaining pleasing image quality compared to other common image restoration approaches.

Dual-supervised attention network for deep cross-modal hashing

Cross-modal hashing has received intensive attention due to its low computation and storage efficiency in cross-modal retrieval task. Most previous cross-modal hashing methods mainly focus on extracting correlated binary codes from the pairwise label, but largely ignore the semantic categories of cross-modal data. On the other hand, human perception exploits category information to connect cross-modal samples. Inspired by this fact, we propose to embed category information into hash codes. More specifically, we introduce semantic prediction loss into our framework to enhance hash codes with category supervision. In addition, there always exists a large gap between features from different modalities (e.g. text and images), leading cross-modal hashing to link irrelevant features for retrieval task. To address this issue, this paper proposes Dual-Supervised Attention Network for Deep Hashing (DSADH) to learn the cross-modal relationship via an elaborately-designed attention mechanism. Our cross-modal network applies cross-modal attention block to efficiently encode rich and relevant features to learn compact hash codes. Extensive experiments on three challenging benchmarks demonstrate that our proposed method significantly improves the retrieval results.

Predict and redirect: Prediction errors support children's word learning.

According to prediction-based learning theories, erroneous predictions support learning. However, empirical evidence for a relation between prediction error and children's language learning is currently lacking. Here we investigated whether and how prediction errors influence children's learning of novel words. We hypothesized that word learning would vary as a function of 2 factors: the extent to which children generate predictions, and the extent to which children redirect attention in response to errors. Children were tested in a novel word learning task, which used eye tracking to measure (a) real-time semantic predictions to familiar referents, (b) attention redirection following prediction errors, and (c) learning of novel referents. Results indicated that predictions and prediction errors interdependently supported novel word learning, via children's efficient redirection of attention. This study provides a developmental evaluation of prediction-based theories and suggests that erroneous predictions play a mechanistic role in children's language learning. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

To autonomously navigate and plan interactions in real-world environments, robots require the ability to robustly perceive and map complex, unstructured surrounding scenes. Besides building an internal representation of the observed scene geometry, the key insight toward a truly functional understanding of the environment is the usage of higher level entities during mapping, such as individual object instances. This work presents an approach to incrementally build volumetric object-centric maps during online scanning with a localized RGB-D camera. First, a per-frame segmentation scheme combines an unsupervised geometric approach with instance-aware semantic predictions to detect both recognized scene elements as well as previously unseen objects. Next, a data association step tracks the predicted instances across the different frames. Finally, a map integration strategy fuses information about their 3D shape, location, and, if available, semantic class into a global volume. Evaluation on a publicly available dataset shows that the proposed approach for building instance-level semantic maps is competitive with state-of-theart methods, while additionally able to discover objects of unseen categories. The system is further evaluated within a real-world robotic mapping setup, for which qualitative results highlight the online nature of the method. Code is available athttps://github.com/ ethz-asl/voxblox-plusplus.

Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding

Is the meaning of an expected stimulus manifest in brain activity even before it appears? Although theories of predictive coding see anticipatory activity as crucial for the understanding of brain function, few studies have explored neurophysiologically manifest semantic predictions. Here, we report predictive negative-going potentials before the onset of action (i.e. whistle and hand clap) and non-action (i.e. pure tone, water drop) sounds. These prediction potentials (PP) indexed the meaning of action-related sounds. Dependent on the body-part-relationship of sound stimuli, neural sources were relatively more prominent in dorsal or ventral motor areas. In contrast, meaningless sounds (pure tones) activated predictive sources in temporal areas close to the auditory cortex; complex environmental sounds induced an anticipatory positivity broadly distributed over the scalp. We also found a systematic relationship between predictive activity and a Mismatch Negativity (MMN) like response to unexpected meaningful words which were presented as rare deviant stimuli amongst frequently repeated sounds. This deviant-elicited potential indexed semantic priming between action sounds and action-related words and semantic mismatch (prediction error). These results suggest a systematic link between perceptual/semantic prediction and matching mechanisms in the processing of sounds and words.

Phonological versus semantic prediction in focus and repair constructions: No evidence for differential predictions

Evidence suggests that the language processing system is predictive. Although past research has established prediction as a general tendency, it is not yet clear whether comprehenders can modulate their anticipatory strategies in response to cues based on sentence constructions. In two visual world eye-tracking experiments, we investigated whether focus constructions (not the hammer but rather the …) and repair disfluencies (the hammer uh I mean the …) would lead listeners to generate different patterns of predictions. In three offline tasks, we observed that participants preferred semantically related continuations (hammer – nail) following focus constructions and phonologically related continuations (hammer – hammock) following disfluencies. However, these offline preferences were not evident in participants’ predictive eye-movements during online language processing: Semantically related (nail) and phonologically related words (hammock) received additional predictive looks regardless of whether the target word appeared in a disfluency or in a focus construction. However, significantly less semantic and phonological activation was observed in two “control” linguistic contexts in which predictive processing was discouraged. These findings suggest that although the prediction system is sensitive to sentence construction, is it not flexible enough to alter the type of prediction generated based on preceding context.

The involvement of subcortical grey matter in verbal semantic comprehension: A systematic review and meta-analysis of fMRI and PET studies

Abstract Semantic processing is a fundamental aspect in human communication. The cortical organization of semantic processing has been exhaustively described, in contrast to inconsistent results on the function of subcortical grey matter structures. Hence, this manuscript reports a systematic review and meta-analysis on the subcortical involvement in verbal semantic comprehension in healthy individuals. The 50 included studies indicate specific contributions by the cerebellum, thalamus and caudate nucleus respectively. In particular, the right posterior cerebellum is proposed to be involved in a semantic executive system and in adequate decision-making. The left thalamus, regulated by the caudate nucleus, might function as a subcortical hub, controlling the access and integration of cortically organized semantic features. Furthermore, a contribution of the cerebellum, thalamus and caudate nucleus in semantic prediction generation and evaluation at sentence level is preliminarily suggested. More research is required to gain insights into the role of the putamen, globus pallidus and subthalamic nucleus.

Research on Semantic Prediction Analysis of Tibetan Text Based on Word2Vec

This article uses Google’s open source Word2Vec tool to input the corpus of the Tibetan text “Sage wedding” after the word segmentation. The words are mapped to a K-dimensional space in the text and transformed into word vectors by using the context information of the vocabulary. The Word2Vec tool then learns to get a vector model, each of which is represented by a unique word vector. A vocabulary is constructed through training text data and then the vector is represented by learning the words. Word vectors capture the laws of many languages, which results the similarity of the distance between words and words. The experimental results show that the accuracy and recall rate based on the Word2Vec training model are very high.

Temporal Dynamics of Syntactic and Semantic Prediction

Temporal Dynamics of Syntactic and Semantic Prediction

Neural evidence for Bayesian trial-by-trial adaptation on the N400 during semantic priming

When semantic information is activated by a context prior to new bottom-up input (i.e. when a word is predicted), semantic processing of that incoming word is typically facilitated, attenuating the amplitude of the N400 event related potential (ERP) – a direct neural measure of semantic processing. N400 modulation is observed even when the context is a single semantically related “prime” word. This so-called “N400 semantic priming effect” is sensitive to the probability of encountering a related prime-target pair within an experimental block, suggesting that participants may be adapting the strength of their predictions to the predictive validity of their broader experimental environment. We formalize this adaptation using a Bayesian learning model that estimates and updates the probability of encountering a related versus an unrelated prime-target pair on each successive trial. We found that our model’s trial-by-trial estimates of target word probability accounted for significant variance in trial-by-trial N400 amplitude. These findings suggest that Bayesian principles contribute to how comprehenders adapt their semantic predictions to the statistical structure of their broader environment, with implications for the functional significance of the N400 component and the predictive nature of language processing.

Ahead of time: Early sentence slow cortical modulations associated to semantic prediction

According to prediction-based accounts of language comprehension, incoming contextual information is constantly used to guide the pre-activation of the most probable continuations to the unfolding sentences. However, there is still scarce evidence of the build-up of these predictions during sentence comprehension. Using event-related brain potentials, we investigated sustained processes associated to semantic prediction during online sentence comprehension. To address this, participants read sentences with varying levels of contextual constraint one word at a time. A 1000 ms interval preceded the final word, which could be congruent or incongruent. A slow sustained negativity developed gradually over the course of sentences, showing differences across conditions, with increasingly larger amplitudes for high than low levels of constraint. The effect was maximal in the interval preceding the closing word. This interval elicited a left-dominant slow negative potential with a graded amplitude modulation to contextual constraint, replicating previous results in speech comprehension. We argue that these slow potentials index the engagement of cognitive operations associated to semantic prediction. In addition, we replicated the finding of an earlier onset of the N400 effect (incongruent minus congruent) for high relative to low contextual constraint, suggesting facilitated processing for contextually-supported and highly expected words. Altogether, these results are consistent with prediction-based models of language comprehension and they also strengthen the value of investigating slow components as potential indices of mechanisms linked to language prediction.

Category-Level Adversaries for Semantic Domain Adaptation

Recent advances in deep learning, especially deep convolutional neural networks, have led to great performance improvement over semantic segmentation systems. Unfortunately, training deep neural networks (DNNs) requires a humongous amount of labeled data, which is laborious and costly to collect and annotate. Thus, plenty of works have proposed an alternative solution to ease the training set creation by using synthetic data. However, models trained on these kinds of data usually under-perform on real images due to the well-known issue of domain shift. To address it, we propose a generative adversarial network (GAN)-based framework called category-level adversarial adaptation networks (CAA-Nets) for domain adaptation in the context of semantic segmentation. Considering semantic predictions that contain spatial and structure information of images, our idea is to make use of this character by imposing discriminators on the semantic predictions. Different from existing works, the proposed framework utilizes a category-level discriminator in the output space to shrink the gap between real and synthetic images. Similar to reinforcement learning, we take final results as a guide to update parameters in the right direction by use of the output-based discriminator. Moreover, to further enhance the performance, we construct an image-based generator and discriminator pair to distill the feature representations obtained by a DNN. Taking advantage of these modules, our model can achieve competitive performance compared with some existing methods. To showcase the generality and scalability of our approach, we evaluate the proposed method on the Cityscapes dataset by adapting from GTAV and SYNTHIA datasets, where the results demonstrate the effectiveness of our method.

Modular reconfiguration of an auditory control brain network supports adaptive listening behavior

Speech comprehension in noisy, multitalker situations poses a challenge. Successful behavioral adaptation to a listening challenge often requires stronger engagement of auditory spatial attention and context-dependent semantic predictions. Human listeners differ substantially in the degree to which they adapt behaviorally and can listen successfully under such circumstances. How cortical networks embody this adaptation, particularly at the individual level, is currently unknown. We here explain this adaptation from reconfiguration of brain networks for a challenging listening task (i.e., a linguistic variant of the Posner paradigm with concurrent speech) in an age-varying sample of n = 49 healthy adults undergoing resting-state and task fMRI. We here provide evidence for the hypothesis that more successful listeners exhibit stronger task-specific reconfiguration (hence, better adaptation) of brain networks. From rest to task, brain networks become reconfigured toward more localized cortical processing characterized by higher topological segregation. This reconfiguration is dominated by the functional division of an auditory and a cingulo-opercular module and the emergence of a conjoined auditory and ventral attention module along bilateral middle and posterior temporal cortices. Supporting our hypothesis, the degree to which modularity of this frontotemporal auditory control network is increased relative to resting state predicts individuals' listening success in states of divided and selective attention. Our findings elucidate how fine-tuned cortical communication dynamics shape selection and comprehension of speech. Our results highlight modularity of the auditory control network as a key organizational principle in cortical implementation of auditory spatial attention in challenging listening situations.