Natural Sentences Research Articles

Vocabulary Size Influences Spontaneous Speech in Native Language Users: Validating the Use of Automatic Speech Recognition in Individual Differences Research.

Previous research has shown that vocabulary size affects performance on laboratory word production tasks. Individuals who know many words show faster lexical access and retrieve more words belonging to pre-specified categories than individuals who know fewer words. The present study examined the relationship between receptive vocabulary size and speaking skills as assessed in a natural sentence production task. We asked whether measures derived from spontaneous responses to everyday questions correlate with the size of participants' vocabulary. Moreover, we assessed the suitability of automatic speech recognition (ASR) for the analysis of participants' responses in complex language production data. We found that vocabulary size predicted indices of spontaneous speech: individuals with a larger vocabulary produced more words and had a higher speech-silence ratio compared to individuals with a smaller vocabulary. Importantly, these relationships were reliably identified using manual and automated transcription methods. Taken together, our results suggest that spontaneous speech elicitation is a useful method to investigate natural language production and that automatic speech recognition can alleviate the burden of labor-intensive speech transcription.

FREGE’S PUZZLE IS HERE TO STAY: TRIVIALITY AND INFORMATIVITY IN NATURAL LANGUAGES

Abstract Frege’s puzzling remarks on the beginning of On Sense and Reference challenge us to explain how true identity sentences of the form a = a can differ in cognitive value from sentences of the form a = b when they are made true by the same object’s self-identity. Some philosophers (e.g. Almog, Glezakos and Paganini) suggest that the puzzle cannot be set up in the context of natural languages since natural sentences, unlike those of regimented formal ones, do not wear their logical properties on their sleeves. In this paper we argue that, on the contrary, there exists a notion of coordination between names which is apt to track the relevant logical properties of natural sentences and therefore to set up the puzzle in natural languages. Frege’s puzzle is here to stay.

Informational masking of speech by acoustically similar intelligible and unintelligible interferers.

Masking experienced when target speech is accompanied by a single interfering voice is often primarily informational masking (IM). IM is generally greater when the interferer is intelligible than when it is not (e.g., speech from an unfamiliar language), but the relative contributions of acoustic-phonetic and linguistic interference are often difficult to assess owing to acoustic differences between interferers (e.g., different talkers). Three-formant analogues (F1+F2+F3) of natural sentences were used as targets and interferers. Targets were presented monaurally either alone or accompanied contralaterally by interferers from another sentence (F0 = 4 semitones higher); a target-to-masker ratio (TMR) between ears of 0, 6, or 12 dB was used. Interferers were either intelligible or rendered unintelligible by delaying F2 and advancing F3 by 150 ms relative to F1, a manipulation designed to minimize spectro-temporal differences between corresponding interferers. Target-sentence intelligibility (keywords correct) was 67% when presented alone, but fell considerably when an unintelligible interferer was present (49%) and significantly further when the interferer was intelligible (41%). Changes in TMR produced neither a significant main effect nor an interaction with interferer type. Interference with acoustic-phonetic processing of the target can explain much of the impact on intelligibility, but linguistic factors-particularly interferer intrusions-also make an important contribution to IM.

Automatic Image Captioning Using Convolution Neural Networks and LSTM

PC vision has turned out to be universal in our general public, with applications in a few fields. Given a lot of pictures, with its inscription, make a prescient model which produces regular, inventive, and intriguing subtitles for the concealed picture. A speedy look at a picture is adequate for a human to call attention to and portray a monstrous measure of insights regarding the visual scene. To rearrange the current issue of producing inscriptions for pictures by making a model which would give exact subtitles to these pictures which can be additionally utilized in other helpful applications and use cases. Be that as it may, this momentous capacity has ended up being a tricky errand for our visual acknowledgment models. Most of the past research in scene acknowledgment has concentrated on naming pictures with a predetermined arrangement of visual classifications and extraordinary advancement has been accom-plished in these undertakings. For a question picture, the past strategies recover pertinent hopeful normal language states by outwardly contrasting the inquiry picture with database pictures. In any case, while shut vocabularies of visual ideas comprise a helpful demonstrating suspicion, they are boundlessly prohibitive when contrasted with the colossal measure of rich depictions that a human can form. These methodologies forced a breaking point on the assortment of inscriptions produced. The model ought to be exempt of suppositions regarding explicit pre decided formats, standards or classes and rather depend on figuring out how to create sentences from the preparation information. The model proposed utilizes Convolution Neural Networks which help to separate highlights of the picture whose subtitle is to be created and afterward by utilizing a probabilistic methodology and Natural Language Processing Techniques reasonable sentences are framed and inscriptions are produced.

LAM: Remote Sensing Image Captioning with Label-Attention Mechanism

Significant progress has been made in remote sensing image captioning by encoder-decoder frameworks. The conventional attention mechanism is prevalent in this task but still has some drawbacks. The conventional attention mechanism only uses visual information about the remote sensing images without considering using the label information to guide the calculation of attention masks. To this end, a novel attention mechanism, namely Label-Attention Mechanism (LAM), is proposed in this paper. LAM additionally utilizes the label information of high-resolution remote sensing images to generate natural sentences to describe the given images. It is worth noting that, instead of high-level image features, the predicted categories’ word embedding vectors are adopted to guide the calculation of attention masks. Representing the content of images in the form of word embedding vectors can filter out redundant image features. In addition, it can also preserve pure and useful information for generating complete sentences. The experimental results from UCM-Captions, Sydney-Captions and RSICD demonstrate that LAM can improve the model’s performance for describing high-resolution remote sensing images and obtain better S m scores compared with other methods. S m score is a hybrid scoring method derived from the AI Challenge 2017 scoring method. In addition, the validity of LAM is verified by the experiment of using true labels.

WAAC: An End-to-End Web API Automatic Calls Approach for Goal-Oriented Intelligent Services

Web API recommendations have recently been studied extensively. However, recommending an API for a service is different than service intelligence. Web API automatic calls are widely used in question–answer dialog applications and service-composed workflow systems to achieve intelligent services. To finish an automatic Web API call not only requires the Web API ID, but also its input parameters. In this paper, we propose an end-to-end Web API automatic calls approach, named WAAC, that translates a goal’s natural language sentences directly to the Web API invoking sequences including its ID and parameters. This end-to-end approach based on the seq2seq encoder–decoder framework, adopts character-level RNN for the Chinese sentences and introduces a copying mechanism to retrieve API parameters. To train the network, a Chinese version dataset of over 1 million natural sentences and API invoking sequence pairs are generated with some manually labeled data and 72 real Web API invoking logs. Experiments obtain a 96% precision on predicting API invoking sequences and show that the character-level RNN and copying mechanism both contribute considerably to achieving a high precision Web API automatic call system for goal-oriented services.

An Integrated Neural Decoder of Linguistic and Experiential Meaning.

The brain is thought to combine linguistic knowledge of words and nonlinguistic knowledge of their referents to encode sentence meaning. However, functional neuroimaging studies aiming at decoding language meaning from neural activity have mostly relied on distributional models of word semantics, which are based on patterns of word co-occurrence in text corpora. Here, we present initial evidence that modeling nonlinguistic "experiential" knowledge contributes to decoding neural representations of sentence meaning. We model attributes of peoples' sensory, motor, social, emotional, and cognitive experiences with words using behavioral ratings. We demonstrate that fMRI activation elicited in sentence reading is more accurately decoded when this experiential attribute model is integrated with a text-based model than when either model is applied in isolation (participants were 5 males and 9 females). Our decoding approach exploits a representation-similarity-based framework, which benefits from being parameter free, while performing at accuracy levels comparable with those from parameter fitting approaches, such as ridge regression. We find that the text-based model contributes particularly to the decoding of sentences containing linguistically oriented "abstract" words and reveal tentative evidence that the experiential model improves decoding of more concrete sentences. Finally, we introduce a cross-participant decoding method to estimate an upper bound on model-based decoding accuracy. We demonstrate that a substantial fraction of neural signal remains unexplained, and leverage this gap to pinpoint characteristics of weakly decoded sentences and hence identify model weaknesses to guide future model development.SIGNIFICANCE STATEMENT Language gives humans the unique ability to communicate about historical events, theoretical concepts, and fiction. Although words are learned through language and defined by their relations to other words in dictionaries, our understanding of word meaning presumably draws heavily on our nonlinguistic sensory, motor, interoceptive, and emotional experiences with words and their referents. Behavioral experiments lend support to the intuition that word meaning integrates aspects of linguistic and nonlinguistic "experiential" knowledge. However, behavioral measures do not provide a window on how meaning is represented in the brain and tend to necessitate artificial experimental paradigms. We present a model-based approach that reveals early evidence that experiential and linguistically acquired knowledge can be detected in brain activity elicited in reading natural sentences.

Data Augmentation for Spoken Language Understanding via Joint Variational Generation

Data scarcity is one of the main obstacles of domain adaptation in spoken language understanding (SLU) due to the high cost of creating manually tagged SLU datasets. Recent works in neural text generative models, particularly latent variable models such as variational autoencoder (VAE), have shown promising results in regards to generating plausible and natural sentences. In this paper, we propose a novel generative architecture which leverages the generative power of latent variable models to jointly synthesize fully annotated utterances. Our experiments show that existing SLU models trained on the additional synthetic examples achieve performance gains. Our approach not only helps alleviate the data scarcity issue in the SLU task for many datasets but also indiscriminately improves language understanding performances for various SLU models, supported by extensive experiments and rigorous statistical testing.

Localizing Natural Language in Videos

In this paper, we consider the task of natural language video localization (NLVL): given an untrimmed video and a natural language description, the goal is to localize a segment in the video which semantically corresponds to the given natural language description. We propose a localizing network (LNet), working in an end-to-end fashion, to tackle the NLVL task. We first match the natural sentence and video sequence by cross-gated attended recurrent networks to exploit their fine-grained interactions and generate a sentence-aware video representation. A self interactor is proposed to perform crossframe matching, which dynamically encodes and aggregates the matching evidences. Finally, a boundary model is proposed to locate the positions of video segments corresponding to the natural sentence description by predicting the starting and ending points of the segment. Extensive experiments conducted on the public TACoS and DiDeMo datasets demonstrate that our proposed model performs effectively and efficiently against the state-of-the-art approaches.

Understanding Pictograph with Facial Features: End-to-End Sentence-Level Lip Reading of Chinese

With the breakthrough of deep learning, lip reading technologies are under extraordinarily rapid progress. It is well-known that Chinese is the most widely spoken language in the world. Unlike alphabetic languages, it involves more than 1,000 pronunciations as Pinyin, and nearly 90,000 pictographic characters as Hanzi, which makes lip reading of Chinese very challenging. In this paper, we implement visual-only Chinese lip reading of unconstrained sentences in a two-step end-to-end architecture (LipCH-Net), in which two deep neural network models are employed to perform the recognition of Pictureto-Pinyin (mouth motion pictures to pronunciations) and the recognition of Pinyin-to-Hanzi (pronunciations to texts) respectively, before having a jointly optimization to improve the overall performance. In addition, two modules in the Pinyin-to-Hanzi model are pre-trained separately with large auxiliary data in advance of sequence-to-sequence training to make the best of long sequence matches for avoiding ambiguity. We collect 6-month daily news broadcasts from China Central Television (CCTV) website, and semi-automatically label them into a 20.95 GB dataset with 20,495 natural Chinese sentences. When trained on the CCTV dataset, the LipCH-Net model outperforms the performance of all stateof-the-art lip reading frameworks. According to the results, our scheme not only accelerates training and reduces overfitting, but also overcomes syntactic ambiguity of Chinese which provides a baseline for future relevant work.

The contribution of audiovisual speech to lexical-semantic processing in natural spoken sentences

ABSTRACT In everyday communication, natural spoken sentences are expressed in a multisensory way through auditory signals and speakers’ visible articulatory gestures. An important issue is to know whether audiovisual speech plays a main role in the linguistic encoding of an utterance until access to meaning. To this end, we conducted an event-related potential experiment during which participants listened passively to spoken sentences and a lexical recognition task. The results revealed that N200 and N400 waves had a greater amplitude after semantically incongruous words than after expected words. This effect of semantic congruency was increased over N200 in the audiovisual trials. Words presented audiovisually also elicited a reduced amplitude of the N400 wave and a facilitated recovery in memory. Our findings shed light on the influence of audiovisual speech on the understanding of natural spoken sentences by acting on the early stages of word recognition in order to access a lexical-semantic network.

Rich Visual and Language Representation with Complementary Semantics for Video Captioning

It is interesting and challenging to translate a video to natural description sentences based on the video content. In this work, an advanced framework is built to generate sentences with coherence and rich semantic expressions for video captioning. A long short term memory (LSTM) network with an improved factored way is first developed, which takes the inspiration of LSTM with a conventional factored way and a common practice to feed multi-modal features into LSTM at the first time step for visual description. Then, the incorporation of the LSTM network with the proposed improved factored way and un-factored way is exploited, and a voting strategy is utilized to predict candidate words. In addition, for robust and abstract visual and language representation, residuals are employed to enhance the gradient signals that are learned from the residual network (ResNet), and a deeper LSTM network is constructed. Furthermore, three convolutional neural network based features extracted from GoogLeNet, ResNet101, and ResNet152, are fused to catch more comprehensive and complementary visual information. Experiments are conducted on two benchmark datasets, including MSVD and MSR-VTT2016, and competitive performances are obtained by the proposed techniques as compared to other state-of-the-art methods.

Cross-domain personalized image captioning

Image captioning aims to translate an image to a complete and natural sentence. It involves both computer vision and natural language processing. Though image captioning has achieved good results under the rapid development of deep neural networks, excessively pursuing the evaluation results of the captioning models makes the generated text description too conservative in practical applications. It is necessary to increase the diversity of the text description and account for prior knowledge such as the user’s favorite vocabularies and writing styles. In this paper, we study the personalized image captioning which can generate sentences to describe the user’s own story and feelings of life with the most preferred word expression. Moreover, we propose cross-domain personalized image captioning (CDPIC) to learn domain-invariant captioning models which can be applied on different social media platforms. The proposed method can flexibly model user interest by embedding the user ID as an interest vector. To the best of our knowledge, we propose the first cross-domain personalized image captioning approach by combining the user interest modeling and a simple and effective domain-invariant constraint. The effectiveness of the proposed method is verified on datasets from the Instagram and Lookbook platforms.

Coordination patterns in Essential Tremor patients with Deep Brain Stimulation: Syllables with low and high complexity

This study investigated syllable coordination patterns in Essential Tremor (ET) patients treated with Deep Brain Stimulation (DBS) by using electromagnetic articulography. We analyzed articulatory timing patterns for nine ET patients with activated and inactivated DBS and compared them to a group of healthy age-matched controls. We focused on timing patterns among gestures in syllables with low and high complexity in natural sentence production (simple CV versus complex CCV syllables). These articulatory patterns were interpreted in the framework of a coupled oscillator planning model of speech timing. In simple CV syllables, ET patients did show a similar coordination pattern to healthy control speakers. However, when complexity increased, the patients showed deviant coordination patterns for complex CCV syllables. These deviant patterns even aggravated under activated stimulation. We were able to show that the behavior of the speech system changes when the stimulation was activated, inducing a change in the dynamical system the ET patients have to adapt to. We conclude that coordination problems are not categorical but gradient in nature, pointing to the fact that there are dynamic mechanisms of regulation behind phonetic realization of phonological syllable parses.

Natural-language-based intelligent retrieval engine for BIM object database

Rapid growth of building components in the BIM object database increases the difficulty of the efficient query of components that users require. Retrieval technology such as Autodesk Seek in America and BIMobject in Europe, which are widely used in BIM databases, are unable to understand what the search field truly means, causing a lack of completion and a low accuracy rate for results incapable of meeting the demands of users. To tackle such a problem, this paper puts forward a natural-language-based intelligent retrieval engine for the BIM object database and Revit modeling. First, a domain ontology is constructed for semantic understanding, and the BIM object database framework is established for testing our search engine. Second, “target keyword” and “restriction sequence” proposed are extracted from the natural sentences of users. Then, a final query is formed, combining concepts of “keyword” and “restriction sequence”, and its concepts are expanded through the semantic relationship in ontology. Finally, the results are presented after mapping from the final query to the BIM object database and ranking of results. Compared with traditional keyword-based methods, the experimental results demonstrate that our method outperforms the traditional methods.

Dichotic integration of acoustic-phonetic information: Competition from extraneous formants increases the effect of second-formant attenuation on intelligibility.

Differences in ear of presentation and level do not prevent effective integration of concurrent speech cues such as formant frequencies. For example, presenting the higher formants of a consonant-vowel syllable in the opposite ear to the first formant protects them from upward spread of masking, allowing them to remain effective speech cues even after substantial attenuation. This study used three-formant (F1+F2+F3) analogues of natural sentences and extended the approach to include competitive conditions. Target formants were presented dichotically (F1+F3; F2), either alone or accompanied by an extraneous competitor for F2 (i.e., F1±F2C+F3; F2) that listeners must reject to optimize recognition. F2C was created by inverting the F2 frequency contour and using the F2 amplitude contour without attenuation. In experiment 1, F2C was always absent and intelligibility was unaffected until F2 attenuation exceeded 30 dB; F2 still provided useful information at 48-dB attenuation. In experiment 2, attenuating F2 by 24 dB caused considerable loss of intelligibility when F2C was present, but had no effect in its absence. Factors likely to contribute to this interaction include informational masking from F2C acting to swamp the acoustic-phonetic information carried by F2, and interaural inhibition from F2C acting to reduce the effective level of F2.

Informational masking of speech analogues by intelligible and non-intelligible but acoustically similar interferers

Informational masking of target speech is generally greater when the interfering speech is intelligible than when it is not (e.g., speech from an unfamiliar language), but the relative contributions of acoustic-phonetic and linguistic interference are often difficult to assess owing to acoustic differences between interferers (e.g., different talkers). This study used three-formant analogues (F1 + F2 + F3) of natural sentences as targets and interferers. Target formants were presented monaurally (F0 = 120.3 Hz) either alone or accompanied with the contralateral ear by interfering formants from another sentence (F0 = 151.5 Hz); a target-to-masker ratio (TMR) between ears of 0, 6, or 12 dB was used. Interferers were either intelligible or rendered non-intelligible by delaying F2 and advancing F3 by 150 ms relative to F1, a manipulation designed to minimize spectro-temporal differences between corresponding interferers. Target-sentence intelligibility (keywords correct) was 67% when presented alone but fell considerably when a non-intelligible interferer was present (49%) and significantly further when the interferer was intelligible (41%). The changes in TMR produced neither a significant main effect nor an interaction with interferer type. The results suggest that although linguistic factors contribute to informational masking, interference with acoustic-phonetic processing of the target can explain much of the impact on intelligibility. [Work supported by ESRC.]

Morphological Verb-Aware Tibetan Language Model

The Tibetan language model (TLM) is the key to Tibetan natural language processing. In this paper, we first observe that, different from widely used languages, Tibetan contains many morphological verbs that rarely appear in natural sentences but play a key role in accurate text prediction. This property is usually ignored by existing methods and makes traditional training strategies less effective in constructing accurate and robust TLMs. Hence, we propose a morphological verb-aware TLM by offline learning via a character frequency reweighting strategy and online tuning of discriminative weights conditioned on morphological verbs. However, because of the influence of morphological verbs on the tense and semantics of sentences, it is necessary to consider the morphological verbs in Tibetan. As a result, compared with state-of-the-art methods, our method not only reduces the perplexity but also improves the character error on tasks of the text prediction and automatic speech recognition (ASR).

Semantic anomaly detection in school-aged children during natural sentence reading - A study of fixation-related brain potentials.

In this study, we investigated the effects of context-related semantic anomalies on the fixation-related brain potentials of 12–13-year-old Finnish children in grade 6 during sentence reading. The detection of such anomalies is typically reflected in the N400 event-related potential. We also examined whether the representation invoked by the sentence context extends to the orthographic representation level by replacing the final words of the sentence with an anomalous word neighbour of a plausible word. The eye-movement results show that the anomalous word neighbours of plausible words cause similar first-fixation and gaze duration reactions, as do other anomalous words. Similarly, we observed frontal negativity in the fixation-related potential of the unrelated anomalous words and in the anomalous word neighbours. This frontal negativity was larger in both anomalous conditions than in the response elicited by the plausible condition. We thus show that the brain successfully uses context to separate anomalous words from plausible words on a single letter level during free reading. From the P600 response of the scalp waveform, we observed that the P600 was delayed in the anomalous word neighbour condition. We performed group-level decomposition on the data with ICA (independent component analysis) and analysed the time course and source structure of the decomposed data. This analysis of decomposed brain signals not only confirmed the delay of the P600 response but also revealed that the frontal negativity concealed s more typical and separate N400 response, which was similarly delayed in the anomalous word neighbour condition, as was the P600 response. Source analysis of these independent components implicated the right frontal eye field as the cortical source for the frontal negativity and the middle temporal and parietal regions as cortical sources for the components resembling the N400 and P600 responses. We interpret the delays present in N400 and P600 responses to anomalous word neighbours to reflect competition with the representation of the plausible word just one letter different.

Deep Audio-Visual Speech Recognition.

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.