Natural Speech Processing Research Articles

Machine Learning Approaches to Analyze Speech-Evoked Neurophysiological Responses.

Purpose Speech-evoked neurophysiological responses are often collected to answer clinically and theoretically driven questions concerning speech and language processing. Here, we highlight the practical application of machine learning (ML)-based approaches to analyzing speech-evoked neurophysiological responses. Method Two categories of ML-based approaches are introduced: decoding models, which generate a speech stimulus output using the features from the neurophysiological responses, and encoding models, which use speech stimulus features to predict neurophysiological responses. In this review, we focus on (a) a decoding model classification approach, wherein speech-evoked neurophysiological responses are classified as belonging to 1 of a finite set of possible speech events (e.g., phonological categories), and (b) an encoding model temporal response function approach, which quantifies the transformation of a speech stimulus feature to continuous neural activity. Results We illustrate the utility of the classification approach to analyze early electroencephalographic (EEG) responses to Mandarin lexical tone categories from a traditional experimental design, and to classify EEG responses to English phonemes evoked by natural continuous speech (i.e., an audiobook) into phonological categories (plosive, fricative, nasal, and vowel). We also demonstrate the utility of temporal response function to predict EEG responses to natural continuous speech from acoustic features. Neural metrics from the 3 examples all exhibit statistically significant effects at the individual level. Conclusion We propose that ML-based approaches can complement traditional analysis approaches to analyze neurophysiological responses to speech signals and provide a deeper understanding of natural speech and language processing using ecologically valid paradigms in both typical and clinical populations.

Sequence Segmentation Using Semi-Markov Conditional Random Fields

Many applications in natural language, speech processing and data integration require model-based segmentation of sequences. Semi-Markov conditional random fields (semi-CRFs) are a generalization of CRFs and provide a full conditional distribution over all possible segmentation of a sequence. Semi-CRFs are particularly suitable for tasks that entail segment-level features such as match with existing dictionary of segments. Empirical results on real-life NER tasks show that they yield higher accuracy than CRFs, but the straightforward foreword–backward inference algorithm requires 3–10 times the computation cost of CRFs. This running time can be reduced significantly by exploiting overlapping features across segments. We present a succinct representation of overlapping features and an efficient training algorithm that can sum over all possible input segmentation in time that is sub-quadratic in the input length, even while imposing no bound on the maximum segment length. Consequently, the running time becomes comparable to CRFs even with the addition of useful entity-level features on large input segments.

Using perceptually relevant speech stimuli for making physiological measurements of the human auditory brainstem

Speech perception is one of the most important functions of the auditory system. The brainstem is an essential part of this process. While studies of natural speech processing typically employ behavioral tasks, and more recently cortical electroencephalography, studies of the human brainstem have been limited (by necessity) to short stimuli like clicks, tonebursts, and single syllables. We recently described a method for presenting continuous naturally uttered speech and deriving the auditory brainstem response. This method makes it possible to create engaging tasks using natural speech while making simultaneous physiological measurements, with applications to a wide range of scientific questions. One such question, with a history of mixed findings, is that of selective attention’s role in brainstem processing. We will discuss our work using the speech-derived auditory brainstem response in a two-talker listening task. In keeping with history, our results seem to differ from those of other recent studies using a similar technique. We will also discuss preliminary work at adapting the technique for audiological purposes, in hopes that using speech stimuli will provide a more accurate clinical predictor of speech perception.

How Artificial Intelligence and Machine Learning May Eventually Change Content Creation Methodologies

For almost 30 years, digital nonlinear editing systems (DNLEs) have functioned as a practical replacement for film and videotape editing. While DNLEs have progressed in their capabilities by offering increased video resolutions and visual effects, they have not fundamentally changed their operational constructs. Editors must still choose in and out points of shots and then methodically edit those shots into a cohesive sequencing. Technology improvements in artificial intelligence and machine learning have the potential to profoundly impact how DNLE systems operate, and, in turn, content creation methodologies will dramatically change. This paper examines the effects of image recognition, natural speech processing, language recognition, cognitive metadata extraction, tonal analysis, and data and statistical integration on the creation of content.

Non-linear Dimensionality Reduction-based Intrusion Detection using Deep Autoencoder

The intrusion detection has become core part of any network of computers due to increasing amount of digital content available. In parallel, the data breaches and malware attacks have also grown in large numbers which makes the role of intrusion detection more essential. Even though many existing techniques are successfully used for detecting intruders but new variants of malware and attacks are being released every day. To counterfeit these new types of attacks, intrusion detection must be designed with state of art techniques such as Deep learning. At present the Deep learning techniques have a strong role in Natural Language Processing, Computer Vision and Speech Processing. This paper is focused on reviewing the role of deep learning techniques for intrusion detection and proposing an efficient deep Auto Encoder (AE) based intrusion detection technique. The intrusion detection is implemented in two stages with a binary classifier and multiclass classification algorithm (dense neural network). The performance of the proposed approach is presented and compared with parallel methods used for intrusion detection. The reconstruction error of the AE model is compared with the PCA and the performance of both anomaly detection and the multiclass classification is analyzed using metrics such as accuracy and false alarm rate. The compressed representation of the AE model helps to lessen the false alarm rate of both anomaly detection and attack classification using SVM and dense NN model respectively.

Evaluating deep learning approaches to characterize and classify the DGAs at scale

In recent years, domain generation algorithms (DGAs) are the foundational mechanisms for many malware families. Mainly, due to the fact that DGA can generate immense number of pseudo random domain names to associate to a command and control (C2) infrastructures. This paper focuses on to detect and classify the pseudo random domain names without relying on the feature engineering or any other linguistic, contextual or semantics and statistical information by adopting deep learning approaches. A deep learning approach is a complex model of traditional machine learning mechanism that has received renewed interest by solving the long-standing tasks in artificial intelligence (AI) related to the field of natural language processing, image recognition, speech processing and many others. They have immense capability to extract optimal feature representations by taking input as in the form of raw input texts. To leverage this and to transfer the performance enhancement in aforementioned areas towards characterize, detect and classify the DGA generated domain names to a specific malware family, this paper adopts deep learning mechanisms with a known one million benign domain names from Alexa, OpenDNS and a corpus of malicious domain names generated from 17 DGA malware families in real time for training in character and bigram level and a trained model has been evaluated on the OSNIT data set in real-time. Specifically, to understand the effectiveness of various deep learning mechanisms, we used recurrent neural network (RNN), identity-recurrent neural network (I-RNN), long short-term memory (LSTM), convolution neural network (CNN), and convolutional neural network-long short-term memory (CNN-LSTM) architectures. Additionally, to find out an optimal architecture, experiments are done with various configurations of network parameters and network structures. All experiments run up to 1000 epochs with a learning rate set in the range [0.01-0.5]. Overall, deep learning approaches, particularly family of recurrent neural network and a hybrid network (where the first layer is CNN and a subsequent layer is LSTM) have showed significant performance with a highest detection rate 0.9945 and 0.9879 respectively. The main reason is deep learning approaches have inherent mechanisms to capture hierarchical feature extraction and long range-dependencies in sequence inputs.

Deep learning for Arabic NLP: A survey

The recent advances in deep learning (DL) have caused breakthroughs in many fields such as computer vision, natural language processing (NLP) and speech processing. Many DL based approaches have been shown to produce state-of-the-art results on various tasks that are of great importance to online social networks (OSN) and social computing such as sentiment analysis (SA) and pharmacovigilance. NLP tasks are becoming very prominent in OSN and DL is offering researchers and practitioners exciting new directions to address these tasks. In this paper, we provide a survey of the published papers on using DL techniques for NLP. We focus on the Arabic language due to its importance, the scarcity of resources on it and the challenges associated with working on it. We notice that DL has yet to receive the attention it deserves from the Arabic NLP (ANLP) community compared with the attention it is getting for other languages despite the vast adoption of social networks in the Arab world. The majority of the early works on using DL for ANLP focused on OCR-related problems while the more recent ones are more diverse with the increasing interest in applying DL to SA, machine translation, diacritization, etc. This survey should serve as a guide for the young and growing ANLP community in order to help bridge the huge gap between ANLP literature and the much richer and more mature English NLP literature.

ScaleDeep

Deep Neural Networks (DNNs) have demonstrated state-of-the-art performance on a broad range of tasks involving natural language, speech, image, and video processing, and are deployed in many real world applications. However, DNNs impose significant computational challenges owing to the complexity of the networks and the amount of data they process, both of which are projected to grow in the future. To improve the efficiency of DNNs, we propose ScaleDeep, a dense, scalable server architecture, whose processing, memory and interconnect subsystems are specialized to leverage the compute and communication characteristics of DNNs. While several DNN accelerator designs have been proposed in recent years, the key difference is that ScaleDeep primarily targets DNN training, as opposed to only inference or evaluation. The key architectural features from which ScaleDeep derives its efficiency are: (i) heterogeneous processing tiles and chips to match the wide diversity in computational characteristics (FLOPs and Bytes/FLOP ratio) that manifest at different levels of granularity in DNNs, (ii) a memory hierarchy and 3-tiered interconnect topology that is suited to the memory access and communication patterns in DNNs, (iii) a low-overhead synchronization mechanism based on hardware data-flow trackers, and (iv) methods to map DNNs to the proposed architecture that minimize data movement and improve core utilization through nested pipelining. We have developed a compiler to allow any DNN topology to be programmed onto ScaleDeep, and a detailed architectural simulator to estimate performance and energy. The simulator incorporates timing and power models of ScaleDeep's components based on synthesis to Intel's 14nm technology. We evaluate an embodiment of ScaleDeep with 7032 processing tiles that operates at 600 MHz and has a peak performance of 680 TFLOPs (single precision) and 1.35 PFLOPs (half-precision) at 1.4KW. Across 11 state-of-the-art DNNs containing 0.65M-14.9M neurons and 6.8M-145.9M weights, including winners from 5 years of the ImageNet competition, ScaleDeep demonstrates 6x-28x speedup at iso-power over the state-of-the-art performance on GPUs.

The right hemisphere is highlighted in connected natural speech production and perception

Current understanding of the cortical mechanisms of speech perception and production stems mostly from studies that focus on single words or sentences. However, it has been suggested that processing of real-life connected speech may rely on additional cortical mechanisms. In the present study, we examined the neural substrates of natural speech production and perception with magnetoencephalography by modulating three central features related to speech: amount of linguistic content, speaking rate and social relevance. The amount of linguistic content was modulated by contrasting natural speech production and perception to speech-like non-linguistic tasks. Meaningful speech was produced and perceived at three speaking rates: normal, slow and fast. Social relevance was probed by having participants attend to speech produced by themselves and an unknown person. These speech-related features were each associated with distinct spatiospectral modulation patterns that involved cortical regions in both hemispheres. Natural speech processing markedly engaged the right hemisphere in addition to the left. In particular, the right temporo-parietal junction, previously linked to attentional processes and social cognition, was highlighted in the task modulations. The present findings suggest that its functional role extends to active generation and perception of meaningful, socially relevant speech.

Instant Diacritics Restoration System for Sindhi Accent Prediction using N-Gram and Memory-Based Learning Approaches

The script of Sindhi Language is highly complex due to many complexities including abundance of homographic words. The interpretation of the text turns so tough due to the possibility of multitudinal meanings associated with a homographic word unless given specific pronunciation with the help of diacritics. Diacritics help the readers to comprehend the text easily. Due to the rapidly developing nature of this era, people don’t bother writing diacritics in routine applications of life. Besides creating difficulties for human reading, the absence of diacritics does also make the text abstruse for machine reading. Relatively alike human, machines may also lead to semantic and syntactic complexities during computational processing of the language. Instant diacritics restoration is an approach emerged from the text prediction systems. This type of diacritics restoration is an unprecedented work in the realm of natural language processing, particularly in Indo-Aryan languages. A proposition for a framework using N-Grams and Memory-Based Learning approach is made in this work. The grab-point of this mechanism is its 99.03% accuracy on the corpus of Sindhi language during the experiments. The comparative edge of instant diacritics restoration is its being source of expedition in the performance of other natural language and speech processing applications. The future development of this approach seems vivid and clear for Sindhi orthography is highly similar to those of Arabic, Urdu, Persian and other languages based on this type of script.

Significant Enhancements in Machine Translation by Various Deep Learning Approaches

A new and transforming technology for natural language processing and speech processing is deep learning. Deep learning extends various operative ways to train computer systems for learning and it gives significant advances for that. If the right system or architecture is developed with deep learning methods then the systems can automatically learn from data itself without the requirement of designing it overtly. This technique of machine learning changes the perspective of addressing natural language and speech technologies considerably. Deep learning was first acquainting with Machine Translation in the standard statistical systems. This paper addresses the progress of introduction of deep learning in machine translation. It describes and includes all the topics like integrating deep learning in statistical machine translation, developing end-to-end neural machine translation systems, introducing deep learning in machine translation evaluation. Several research directions are drawn in terms of how deep learning can influence machine translation.

Language as Sensor in Human‐Centered Computing: Clinical Contexts as Use Cases

Recognizing the increased emphasis on adaptive and personalized computing, as well as the popularity of multimodal natural language and speech processing in human‐centered applications, this paper articulates and discusses the role of natural language as a rich and meaningful sensor of the human body and mind. As the primary means of human interaction, information sharing, and social, emotional, and interpersonal grounding, linguistic sensing provides a particularly useful and rich sensory window for understanding the dynamically and temporally unfolding state of the human body and mind for non‐linguistic aims. For sensory monitoring, linguistic data are unobtrusive, inexpensive, pervasive, natural, especially insightful from the perspective of cognitive processing, and generally convenient to capture, in contrast to many other sensor modalities that require excessive wearable equipment or smart clothing. Using clinical computational linguistics as a springboard for discussing and exemplifying this unconventional yet increasingly prevalent view of language, including its role in practical and serviceable linguistic analysis, this overview surveys recent work in computational linguistics that broadly takes advantage of linguistic sensing's potential. The discussion also outlines existing challenges and considerations for continued research in this area, from the perspective of natural language as a sensor in human‐centered computing and the development of innovative and applied language technologies.

MODELING OF FUNDAMENTAL FREQUENCY CONTOURS FOR THAI DIALECTS WITH LARGE SPEECH DATABASE

In four core regions of Thailand, there are four main dialects including central, north, northeast and south dialects. The prosody is significantly unique for each dialect. One important factor determining the prosody is the fundamental frequency. As a result, modeling of Fundamental frequency (F0) contour is very important for the natural speech processing. Even though there are many modeling techniques for modeling the F0 contour. In this study, the Fujisaki’s model has been selected because of its achievement in modeling of various Thai speech units. This study proposes an analysis of model parameters of Thai speech prosody for four regional dialects and two genders. Seven derived parameters from the Fujisaki’s model are as follows. The first parameter is baseline frequency which is the lowest level of F0 contour. The second and third parameters are the numbers of phrase commands and tone commands which reflect the frequencies of surges of the utterance in global and local levels, respectively. The fourth and fifth parameters are phrase command and tone command durations which reflect the speed of speaking and the length of a syllable, respectively. The sixth and seventh parameters are amplitudes of phrase command and tone command which reflect the energy of the global speech and the energy of local syllable. In the experimental results, the large speech material of each regional dialect includes 50 samples of 50 sentences with male and female speech. It can be obviously seen that most of the proposed parameters can distinguish four kinds of regional dialects explicitly. The results reveal that the proposed parameters of Fujisaki’s model can distinguish the regional dialects explicitly.

Effects of Noises on Fujisaki’s Model of Fundamental Frequency Contours for Thai Dialects

Problem statement: Modeling of fundamental frequency (F0) contour plays an important role on the natural speech processing, since F0 is an important speech feature defining the human speech prosody. In Thai, there are four main dialects spoken by Thai people residing in four core region including central, north, northeast and south regions. Environmental noises are also plays an important role in corrupting the speech quality. The study of effects of noises on modeling of F0 contour for Thai dialects will evaluate robustness of the modeling techniques. Approach: The Fujisaki’s model has been selected in this study because of its achievement in modeling of various Thai speech units. Four types of environmental noises are simulated with different levels of power. The differences among the model parameters of four Thai dialects have been summarized. This study proposes an analysis of model parameters for Thai speech prosody with four regional dialects and two genders and four types of noises. Seven derived parameters from the Fujisaki’s model are as follows. The first parameter is baseline frequency which is the lowest level of F0 contour. The second and third parameters are the numbers of phrase commands and tone commands which reflect the frequencies of surges of the utterance in global and local levels, respectively. The fourth and fifth parameters are phrase command and tone command durations which reflect the speed of speaking and the length of a syllable, respectively. The sixth and seventh parameters are amplitudes of phrase command and tone command which reflect the energy of the global speech and the energy of local syllable. Results: In the experiments, each regional dialect includes 10 samples of 10 sentences with male and female speech. Four types of noises include train, factory, car and air conditioner. Moreover, five levels of each type of noise are varied from 0-20 dB. The results show that most of the proposed parameters can distinguish four kinds of regional dialects explicitly. Conclusion: By using the Fujisaki’s model, the results confirm that the proposed parameters can distinguish the regional dialects efficiently. However, the simulated noises deteriorate the F0 contours and also distort the model parameters.

Augmenting Conversational Environment

People are proficient in collaboratively forming and maintaining gatherings thereby shaping and cultivating collective thoughts through fluent conversational interactions. A big challenge is to develop a technology for augmenting the conversational environment so that people can conduct even better conversational interactions for collective intelligence and creation. Conversational informatics is a field of research that focuses on investigating conversational interactions and designing intelligent artifacts that can augment conversational interactions. The field draws on a foundation provided by artificial intelligence, natural language processing, speech and image processing, cognitive science, and conversation analysis. In this article, the author overviews a methodology for developing augmented conversational environment and major achievements. The author also discusses issues for making agents empathic so that they can induce sustained and constructive engagement with people.

Fujisaki’s Model of Thai’s Fundamental Frequency Contours with Environmental Noises

Problem statement: An important human speech feature is the fundamental frequency (F0) contour which represents the speech prosody. It indicates the naturalness and intelligibility of the speech. Modeling of fundamental frequency contour was an essential procedure in the natural speech processing. In speech communication, environmental noise plays an essential role in damaging the digital communication quality. The study of effects of noises on modeling of F0 contour for standard Thai is conducted. Approach: The selected modeling technique in this study was adapted from Fujisaki’s model, because of its achievement in modeling of various Thai speech units. Four types of environmental noises were recorded for different levels of power. This study was proposed an analysis of some parameters of modeling of Thai speech prosody for two genders and four types of noises. The derived Fujisaki’s model was covered seven parameters including baseline frequency, the numbers of phrase commands and tone commands, phrase command and tone command durations, amplitudes of phrase command and tone command. Results: In the experimental results, the standard Thai of 2 samples of 5 sentences with 5 males and 5 females was used. Four types of noises include train, factory, car and air conditioner. Five levels of each type of noise were varied from 0-20 dB. The results were showing that the different noises give the distinguished effects for most of the proposed model parameters. Conclusion: The results confirm that the effects of four types of noises are significantly different. It can be seen that the environmental noises deteriorate the model parameters empirically.

Forecasting the Experience of Future Entertainment Technology

Advances in gaming and other entertainment technologies are evolving rapidly and create new conceptual challenges for understanding and explaining the user experiences they can facilitate. The present article reports a prospective study on a particularly promising entertainment technology of the future: Interactive storytelling (IS). Integrating various streams of computing technology, such as advanced visualization, natural speech processing, and autonomous agents, IS systems are envisioned to offer new, personalized and thus unique kinds of entertainment to mass audiences of the future. The authors refer to existing models of media entertainment for a theoretical analysis and analyze expert interviews with members of the international IS development community to lay out the foundations for a forecast model of the entertainment experience of future IS systems. The resulting model organizes fundamental requirements, modes of users’ information processing, and specific types of (pleasant) experiences, which holds implications for (future) entertainment theory and research that accompanies further development of IS media.

The Role of Age-Related Neural Timing Variability in Speech Processing

Age-related difficulties in speech processing remain a concern, especially as technology continues to depend heavily on successful speech comprehension on the part of users. Event-related potentials (ERPs) have frequently been used to assess age-related changes in the processing of language. Specifically, the amplitude of the ERP is often compared between conditions or groups of interest. In constructing ERPs, many neurophysiologic responses are averaged together to reduce the contribution of uncorrelated background activity in the electroencephalogram (EEG). However, if variability in the timing of each potential on a trial-by-trial basis (i.e., “latency jitter”) is confounded with a variable of interest, then the presence of amplitude differences observed in the average ERP might not be solely the result of genuine amplitude differences, but also timing variability. We examined the role latency jitter may play in the well-established observation of age-related changes in the processing of natural speech...

Stuttering and Natural Speech Processing of Semantic and Syntactic Constraints on Verbs

Previous findings from event-related brain potentials (ERPs) indicate that adults who stutter (AWS) exhibit processing differences for visually presented linguistic information. This study explores how neural activations for AWS may differ for a linguistic task that does not require preparation for overt articulation or engage the articulatory loop for silent speech. Syntactic and semantic processing constraints were examined in AWS and adults who are normally fluent (AWNF) by assessment of their behavioral performance and ERPs in a natural speech listening task. AWS performed similarly to AWNF in identifying verb-agreement violations and semantic anomalies, but ERPs elicited by syntactic and semantic constraints indicated atypical neural functions for AWS. ERPs of the AWNF displayed an expected N400 for reduced semantic expectations and a typical P600 for verb-agreement violations. In contrast, both N400s and P600s for the semantic and verb-agreement conditions were observed in the ERPs of the AWS. The findings suggest that AWS may engage semantic-syntactic mechanisms more generally for semantic and syntactic processing. These findings converge with earlier studies using visual stimuli to indicate that whereas linguistic abilities are normal in AWS, underlying brain activity mediating some aspects of language processing may function differently.

Listeners modulate temporally selective attention during natural speech processing

Spatially selective attention allows for the preferential processing of relevant stimuli when more information than can be processed in detail is presented simultaneously at distinct locations. Temporally selective attention may serve a similar function during speech perception by allowing listeners to allocate attentional resources to time windows that contain highly relevant acoustic information. To test this hypothesis, event-related potentials were compared in response to attention probes presented in six conditions during a narrative: concurrently with word onsets, beginning 50 and 100 ms before and after word onsets, and at random control intervals. Times for probe presentation were selected such that the acoustic environments of the narrative were matched for all conditions. Linguistic attention probes presented at and immediately following word onsets elicited larger amplitude N1s than control probes over medial and anterior regions. These results indicate that native speakers selectively process sounds presented at specific times during normal speech perception.