Natural Sentences Research Articles

Language-driven synthesis of 3D scenes from scene databases

We introduce a novel framework for using natural language to generate and edit 3D indoor scenes, harnessing scene semantics and text-scene grounding knowledge learned from large annotated 3D scene databases. The advantage of natural language editing interfaces is strongest when performing semantic operations at the sub-scene level, acting on groups of objects. We learn how to manipulate these sub-scenes by analyzing existing 3D scenes. We perform edits by first parsing a natural language command from the user and transforming it into a semantic scene graph that is used to retrieve corresponding sub-scenes from the databases that match the command. We then augment this retrieved sub-scene by incorporating other objects that may be implied by the scene context. Finally, a new 3D scene is synthesized by aligning the augmented sub-scene with the user's current scene, where new objects are spliced into the environment, possibly triggering appropriate adjustments to the existing scene arrangement. A suggestive modeling interface with multiple interpretations of user commands is used to alleviate ambiguities in natural language. We conduct studies comparing our approach against both prior text-to-scene work and artist-made scenes and find that our method significantly outperforms prior work and is comparable to handmade scenes even when complex and varied natural sentences are used.

ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading

We present the Zurich Cognitive Language Processing Corpus (ZuCo), a dataset combining electroencephalography (EEG) and eye-tracking recordings from subjects reading natural sentences. ZuCo includes high-density EEG and eye-tracking data of 12 healthy adult native English speakers, each reading natural English text for 4–6 hours. The recordings span two normal reading tasks and one task-specific reading task, resulting in a dataset that encompasses EEG and eye-tracking data of 21,629 words in 1107 sentences and 154,173 fixations. We believe that this dataset represents a valuable resource for natural language processing (NLP). The EEG and eye-tracking signals lend themselves to train improved machine-learning models for various tasks, in particular for information extraction tasks such as entity and relation extraction and sentiment analysis. Moreover, this dataset is useful for advancing research into the human reading and language understanding process at the level of brain activity and eye-movement.

Image captioning by incorporating affective concepts learned from both visual and textual components

Abstract Automatically generating a natural sentence describing the content of an image has been extensively researched in artificial intelligence recently, and it bridges the gap between computer vision and natural language processing communities. Most of existing captioning frameworks rely heavily on the visual content, while rarely being aware of the sentimental information. In this paper, we introduce the affective concepts to enhance the emotion expressibility of text descriptions. We achieve this goal by composing appropriate emotional concepts to sentences, which is calculated from large-scale visual and textual repositories by learning both content and linguistic modules. We extract visual and textual representations respectively, followed by combining the latent codes of the two components into a low-dimensional subspace. After that, we decode the combined latent representations and finally generate the affective image captions. We evaluate our method on the SentiCap dataset, which was established with sentimental adjective noun pairs, and evaluate the emotional descriptions with several qualitative and human inception metrics. The experimental results demonstrate the capability of our method for analyzing the latent emotion of an image and providing the affective description which caters to human cognition.

How Subjective Is the Subject? A Fresh Look at Grammatical Relations in Mandarin Chinese

This article re-examines the issue of grammatical relations in Mandarin Chinese in light of the results of recent large-scale typological research on grammatical relations (henceforth GRs) worldwide. Specifically, it discusses three syntactic operations and constructions that are cross-linguistically relevant to the definition of grammatical relations, namely relativisation, reflexivisation, and quantifier float. The study adopts a strictly language-internal typological approach and avails itself of natural linguistic data or sentences sanity-checked by native speakers. The aim of this paper is twofold: first, it explores the hypothesis that, in line with various other languages, GRs in Mandarin Chinese are construction-specific. Second, it proposes an alternative approach capable of explaining the conflicting evidence often pointed out in the literature on GRs and subjecthood in Mandarin Chinese.

Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor Cortex

When speaking, we dynamically coordinate movements of our jaw, tongue, lips, and larynx. To investigate the neural mechanisms underlying articulation, we used direct cortical recordings from human sensorimotor cortex while participants spoke natural sentences that included sounds spanning the entire English phonetic inventory. We used deep neural networks to infer speakers' articulator movements from produced speech acoustics. Individual electrodes encoded a diversity of articulatory kinematic trajectories (AKTs), each revealing coordinated articulator movements toward specific vocal tract shapes. AKTs captured a wide range of movement types, yet they could be differentiated by the place of vocal tract constriction. Additionally, AKTs manifested out-and-back trajectories with harmonic oscillator dynamics. While AKTs were functionally stereotyped across different sentences, context-dependent encoding of preceding and following movements during production of the same phoneme demonstrated the cortical representation of coarticulation. Articulatory movements encoded in sensorimotor cortex give rise to the complex kinematics underlying continuous speech production. VIDEO ABSTRACT.

SciTaiL: A Textual Entailment Dataset from Science Question Answering

We present a new dataset and model for textual entailment, derived from treating multiple-choice question-answering as an entailment problem. SciTail is the first entailment set that is created solely from natural sentences that already exist independently ``in the wild'' rather than sentences authored specifically for the entailment task. Different from existing entailment datasets, we create hypotheses from science questions and the corresponding answer candidates, and premises from relevant web sentences retrieved from a large corpus. These sentences are often linguistically challenging. This, combined with the high lexical similarity of premise and hypothesis for both entailed and non-entailed pairs, makes this new entailment task particularly difficult. The resulting challenge is evidenced by state-of-the-art textual entailment systems achieving mediocre performance on SciTail, especially in comparison to a simple majority class baseline. As a step forward, we demonstrate that one can improve accuracy on SciTail by 5% using a new neural model that exploits linguistic structure.

Multiple order semantic relation extraction

In order to get more comprehensive and accurate knowledge from the Semantic Web, it is essential to design an effective method to extract semantic data from the web and documents. However, as a crucial topic of semantic information extraction, relation extraction is a major challenge in knowledge base construction. Existing methods perform poorly on texts in the open domain due to their complex structure; this paper proposes a method named multiple order semantic relation extraction (MOSRE), which applies for multiple orders, a conceptual expression used in formal logistics, to build semantic patterns for extracting information from hybrid unstructured texts in the open domain with deep semantic analyses. Specifically, the proposed method automatically constructs a multiple order semantic tree from complex natural sentences and converts semantic information into a binary structure. Instead of constructing a large amount of pattern sets for comparing binary semantics between entities, MOSRE splits and reconstructs sentences into a strict hierarchical binary structure with combination rules in order to extract as much semantic information as possible. The extracted triples are then processed into several entity relations in the format 〈subject, relational label, object〉 after named entity recognition and refinement. MOSRE is validated by test results on two different datasets, achieving F1 values of 83.8% on SENT500 and 35.5% on KBP.

Across-ear integration of acoustic-phonetic information under competitive conditions: Effects of attenuation increase in the presence of an extraneous formant

Previous research using consonant-vowel syllables where one ear receives the first formant (F1) and the other receives the second and third has shown that dichotic release from masking allows F2 + F3 to remain effective speech cues even after substantial attenuation. This study used three-formant analogues of natural sentences and extended the approach to include competitive conditions. Target formants were presented dichotically (F1 + F3; F2), either alone or accompanied by an extraneous competitor for F2 (i.e., F1±F2C + F3; F2) that listeners must reject to optimize recognition. In experiment 1, F2C was absent and F2 was attenuated in 6-dB steps (range: -6-48 dB). Intelligibility was unaffected until attenuation >30 dB; F2 still provided useful information at 48-dB attenuation. In experiment 2, F2 was attenuated in 12-dB steps (range: 0-36 dB). F2C was created by inverting the F2 frequency contour and using the F2 amplitude contour without attenuation. When F2C was present, 12-dB attenuation was sufficient to cause some loss of intelligibility; the effect was large when attenuation ≥24 dB. This interaction suggests that some mandatory across-ear spectral integration occurs, such that informational masking arising from F2C rapidly swamps the acoustic-phonetic information carried by F2 as its relative level is attenuated. [Work supported by ESRC.]

Synchronization of Electrophysiological Responses with Speech Benefits Syntactic Information Processing.

In auditory neuroscience, electrophysiological synchronization to low-level acoustic and high-level linguistic features is well established-but its functional purpose for verbal information transmission is unclear. Based on prior evidence for a dependence of auditory task performance on delta-band oscillatory phase, we hypothesized that the synchronization of electrophysiological responses at delta-band frequency to the speech stimulus serves to implicitly align neural excitability with syntactic information. The experimental paradigm of our auditory EEG study uniformly distributed morphosyntactic violations across syntactic phrases of natural sentences, such that violations would occur at points differing in linguistic information content. In support of our hypothesis, we found behavioral responses to morphosyntactic violations to increase with decreasing syntactic information content-in significant correlation with delta-band phase, which had synchronized to our speech stimuli. Our findings indicate that rhythmic electrophysiological synchronization to the speech stream is a functional mechanism that may align neural excitability with linguistic information content, optimizing language comprehension.

Symbol Grounding Without Direct Experience: Do Words Inherit Sensorimotor Activation From Purely Linguistic Context?

Theories of embodied cognition assume that concepts are grounded in non-linguistic, sensorimotor experience. In support of this assumption, previous studies have shown that upwards response movements are faster than downwards movements after participants have been presented with words whose referents are typically located in the upper vertical space (and vice versa for downwards responses). This is taken as evidence that processing these words reactivates sensorimotor experiential traces. This congruency effect was also found for novel words, after participants learned these words as labels for novel objects that they encountered either in their upper or lower visual field. While this indicates that direct experience with a word's referent is sufficient to evoke said congruency effects, the present study investigates whether this direct experience is also a necessary condition. To this end, we conducted five experiments in which participants learned novel words from purely linguistic input: Novel words were presented in pairs with real up- or down-words (Experiment 1); they were presented in natural sentences where they replaced these real words (Experiment 2); they were presented as new labels for these real words (Experiment 3); and they were presented as labels for novel combined concepts based on these real words (Experiment 4 and 5). In all five experiments, we did not find any congruency effects elicited by the novel words; however, participants were always able to make correct explicit judgements about the vertical dimension associated to the novel words. These results suggest that direct experience is necessary for reactivating experiential traces, but this reactivation is not a necessary condition for understanding (in the sense of storing and accessing) the corresponding aspects of word meaning.

Image Caption Generation with Part of Speech Guidance

As a fundamental problem in image understanding, image caption generation has attracted much attention from both computer vision and natural language processing communities. In this paper, we focus on how to exploit the structure information of a natural sentence, which is used to describe the content of an image. We discover that the Part of Speech (PoS) tags of a sentence, are very effective cues for guiding the Long Short-Term Memory (LSTM) based word generator. More specifically, given a sentence, the PoS tag of each word is utilized to determine whether it is essential to input image representation into the word generator. Benefiting from such a strategy, our model can closely connect the visual attributes of an image to the word concepts in the natural language space. Experimental results on the most popular benchmark datasets, e.g., Flickr30k and MS COCO, consistently demonstrate that our method can significantly enhance the performance of a standard image caption generation model, and achieve the conpetitive results.

Distorting temporal fine structure by phase shifting and its effects on speech intelligibility and neural phase locking

Envelope (E) and temporal fine structure (TFS) are important features of acoustic signals and their corresponding perceptual function has been investigated with various listening tasks. To further understand the underlying neural processing of TFS, experiments in humans and animals were conducted to demonstrate the effects of modifying the TFS in natural speech sentences on both speech recognition and neural coding. The TFS of natural speech sentences was modified by distorting the phase and maintaining the magnitude. Speech intelligibility was then tested for normal-hearing listeners using the intact and reconstructed sentences presented in quiet and against background noise. Sentences with modified TFS were then used to evoke neural activity in auditory neurons of the inferior colliculus in guinea pigs. Our study demonstrated that speech intelligibility in humans relied on the periodic cues of speech TFS in both quiet and noisy listening conditions. Furthermore, recordings of neural activity from the guinea pig inferior colliculus have shown that individual auditory neurons exhibit phase locking patterns to the periodic cues of speech TFS that disappear when reconstructed sounds do not show periodic patterns anymore. Thus, the periodic cues of TFS are essential for speech intelligibility and are encoded in auditory neurons by phase locking.

On the ambiguity of interaction and nonlinear main effects in a regime of dependent covariates.

The analysis of large experimental datasets frequently reveals significant interactions that are difficult to interpret within the theoretical framework guiding the research. Some of these interactions actually arise from the presence of unspecified nonlinear main effects and statistically dependent covariates in the statistical model. Importantly, such nonlinear main effects may be compatible (or, at least, not incompatible) with the current theoretical framework. In the present literature, this issue has only been studied in terms of correlated (linearly dependent) covariates. Here we generalize to nonlinear main effects (i.e., main effects of arbitrary shape) and dependent covariates. We propose a novel nonparametric method to test for ambiguous interactions where present parametric methods fail. We illustrate the method with a set of simulations and with reanalyses (a) of effects of parental education on their children's educational expectations and (b) of effects of word properties on fixation locations during reading of natural sentences, specifically of effects of length and morphological complexity of the word to be fixated next. The resolution of such ambiguities facilitates theoretical progress.

Video Captioning With Attention-Based LSTM and Semantic Consistency

Recent progress in using long short-term memory (LSTM) for image captioning has motivated the exploration of their applications for video captioning. By taking a video as a sequence of features, an LSTM model is trained on video-sentence pairs and learns to associate a video to a sentence. However, most existing methods compress an entire video shot or frame into a static representation, without considering attention mechanism which allows for selecting salient features. Furthermore, existing approaches usually model the translating error, but ignore the correlations between sentence semantics and visual content. To tackle these issues, we propose a novel end-to-end framework named aLSTMs, an attention-based LSTM model with semantic consistency, to transfer videos to natural sentences. This framework integrates attention mechanism with LSTM to capture salient structures of video, and explores the correlation between multimodal representations (i.e., words and visual content) for generating sentences with rich semantic content. Specifically, we first propose an attention mechanism that uses the dynamic weighted sum of local two-dimensional convolutional neural network representations. Then, an LSTM decoder takes these visual features at time $t$ and the word-embedding feature at time $t$ $-$ 1 to generate important words. Finally, we use multimodal embedding to map the visual and sentence features into a joint space to guarantee the semantic consistence of the sentence description and the video visual content. Experiments on the benchmark datasets demonstrate that our method using single feature can achieve competitive or even better results than the state-of-the-art baselines for video captioning in both BLEU and METEOR.

Foreign Languages Sound Fast: Evidence from Implicit Rate Normalization.

Anecdotal evidence suggests that unfamiliar languages sound faster than one’s native language. Empirical evidence for this impression has, so far, come from explicit rate judgments. The aim of the present study was to test whether such perceived rate differences between native and foreign languages (FLs) have effects on implicit speech processing. Our measure of implicit rate perception was “normalization for speech rate”: an ambiguous vowel between short /a/ and long /a:/ is interpreted as /a:/ following a fast but as /a/ following a slow carrier sentence. That is, listeners did not judge speech rate itself; instead, they categorized ambiguous vowels whose perception was implicitly affected by the rate of the context. We asked whether a bias towards long /a:/ might be observed when the context is not actually faster but simply spoken in a FL. A fully symmetrical experimental design was used: Dutch and German participants listened to rate matched (fast and slow) sentences in both languages spoken by the same bilingual speaker. Sentences were followed by non-words that contained vowels from an /a-a:/ duration continuum. Results from Experiments 1 and 2 showed a consistent effect of rate normalization for both listener groups. Moreover, for German listeners, across the two experiments, foreign sentences triggered more /a:/ responses than (rate matched) native sentences, suggesting that foreign sentences were indeed perceived as faster. Moreover, this FL effect was modulated by participants’ ability to understand the FL: those participants that scored higher on a FL translation task showed less of a FL effect. However, opposite effects were found for the Dutch listeners. For them, their native rather than the FL induced more /a:/ responses. Nevertheless, this reversed effect could be reduced when additional spectral properties of the context were controlled for. Experiment 3, using explicit rate judgments, replicated the effect for German but not Dutch listeners. We therefore conclude that the subjective impression that FLs sound fast may have an effect on implicit speech processing, with implications for how language learners perceive spoken segments in a FL.

Retrieval of Sentence Sequences for an Image Stream via Coherence Recurrent Convolutional Networks.

We propose an approach for retrieving a sequence of natural sentences for an image stream. Since general users often take a series of pictures on their experiences, much online visual information exists in the form of image streams, for which it would better take into consideration of the whole image stream to produce natural language descriptions. While almost all previous studies have dealt with the relation between a single image and a single natural sentence, our work extends both input and output dimension to a sequence of images and a sequence of sentences. For retrieving a coherent flow of multiple sentences for a photo stream, we propose a multimodal neural architecture called coherence recurrent convolutional network (CRCN), which consists of convolutional neural networks, bidirectional long short-term memory (LSTM) networks, and an entity-based local coherence model. Our approach directly learns from vast user-generated resource of blog posts as text-image parallel training data. We collect more than 22 K unique blog posts with 170 K associated images for the travel topics of NYC, Disneyland , Australia, and Hawaii. We demonstrate that our approach outperforms other state-of-the-art image captioning methods for text sequence generation, using both quantitative measures and user studies via Amazon Mechanical Turk.

The Implementation of Natural Semantic Metalanuage and Semantic Field in Language Teaching: A Case Study

This study aimed at developing a model of learning Balinese vocabularies by implementing Natural Semantic Meta-language (NSM) and semantics field theory. This finding was expected to establish a natural atmosphere of Balinese learning based on semantics primes. Semantics primes development on the basis of semantic field can overcome abstractness meaning of certain words. To prove the truth of the theory construction, the researcher modified a research and development which refers to Borg and Gell models. Data were collected before and after the development of model with certain indicators: the rapid in understanding words meaning, skill in using the word in natural sentence structure, and students’ learning creativity. Significance was measured by using t-test formula for correlated samples. Based on statistical analysis, it was proved that the implementation of NSM and semantic field theory was very effective (significant) in learning Balinese vocabularies on students grade 1, 2, and 3. Effectiveness of learning was achieved because the materials adhering to the natural principle of language acquisition and the availability of direct meaning comparison between words were intuitively related. Direct comparison could overcome the cognitive limitations of students grade 1, 2, and 3 in primary school.

Effects of frequency region and number of formants in an interferer on the informational masking of speech

This study explored whether the extent of informational masking depends on the frequency region and number of formants in an interferer. Target formants—monotonized three-formant analogues of natural sentences—were presented monaurally, with the target ear assigned randomly on each trial. Interferers were presented contralaterally. In experiment 1, single-formant interferers were created using the time-reversed F2 frequency contour and constant amplitude, RMS-matched to F2. Interferer center frequency was matched to that of F1, F2, or F3, while maintaining the extent of formant-frequency variation (depth) on a log scale. In experiment 2, the interferer comprised either one formant (F1) or all three, created using the time-reversed frequency contours of the corresponding targets and RMS-matched constant amplitudes. Including the higher formants had little effect on interferer intensity. Interferer formant-frequency variation was scaled to 0%, 50%, or 100% of the original depth. Adding an interferer lowered target intelligibility; frequency region had relatively little effect, but increasing depth of formant-frequency variation and number of formants had independent and additive effects. This suggests that the impact on intelligibility depends primarily on the number of formants and the overall extent of frequency variation in the interferer, not on any particular frequency region. [Work supported by ESRC.]

Center-shared sliding ensemble of neural networks for syntax analysis of natural language

Fixing input sites in ensembles restrict learning movable and distant patterns.The new ensemble changes the sites across component classifiers sharing one position.The ensemble improves accuracy in learning movable patterns and its lower bound.The neural network framework with the ensemble improves accuracy in syntax analysis.The framework is accurate as state-of-the-art methods in Spanish dependency parsing. In this paper, we introduce a new ensemble method specialized to sequential labeling for syntax analysis and propose a neural network framework adopting the ensemble for dependency parsing of natural sentences. The ensemble method assigns sliding input sites to component classifiers which commonly include the position of the label to predict. The method improves labeling accuracy compared to simple ensemble with weighted voting if critical input features have flexible and long distance from the position to predict over sentences. We show the impact of the ensemble through theoretical estimation of its lower bound accuracy and through empirical analysis in a toy problem varying the strength of movability of critical input features. We apply the proposed neural network framework to the two phases of dependency parsing: dependency and relation tagging. Additionally, we newly define the dependency tagging problem using relative dependency and provide a post-processing method to build correct parse trees. In the practical dependency parsing of Spanish IULA corpus, applying the ensemble instead of the simple weighted voting significantly improves accuracy by 0.09% in relation tagging and by 0.06% to 1.59% with respect to the comparison settings in dependency tagging. The framework shows at least 0.28% improvement in the unlabeled attachment score and 0.14% in the labeled attachment score compared to state-of-the-art dependency parsers.

Video Captioning with Listwise Supervision

Automatically describing video content with natural language is a fundamental challenging that has received increasing attention. However, existing techniques restrict the model learning on the pairs of each video and its own sentences, and thus fail to capture more holistically semantic relationships among all sentences. In this paper, we propose to model relative relationships of different video-sentence pairs and present a novel framework, named Long Short-Term Memory with Listwise Supervision (LSTM-LS), for video captioning. Given each video in training data, we obtain a ranking list of sentences w.r.t. a given sentence associated with the video using nearest-neighbor search. The ranking information is represented by a set of rank triplets that can be used to assess the quality of ranking list. The video captioning problem is then solved by learning LSTM model for sentence generation, through maximizing the ranking quality over all the sentences in the list. The experiments on MSVD dataset show that our proposed LSTM-LS produces better performance than the state of the art in generating natural sentences: 51.1% and 32.6% in terms of BLEU@4 and METEOR, respectively. Superior performances are also reported on the movie description M-VAD dataset.