Speech Annotation Research Articles

Psychosocial Features for Hate Speech Detection in Code-switched Texts

This study examines the problem of hate speech identification in codeswitched text from social media using a natural language processing approach. It explores different features in training nine models and empirically evaluates their predictiveness in identifying hate speech in a ~50k human-annotated dataset. The study espouses a novel approach to handle this challenge by introducing a hierarchical approach that employs Latent Dirichlet Analysis to generate topic models that help build a high-level Psychosocial feature set that we acronym PDC. PDC groups similar meaning words in word families, which is significant in capturing codeswitching during the preprocessing stage for supervised learning models. The high-level PDC features generated are based on a hate speech annotation framework [1] that is largely informed by the duplex theory of hate [2]. Results obtained from frequency-based models using the PDC feature on the dataset comprising of tweets generated during the 2012 and 2017 presidential elections in Kenya indicate an f-score of 83% (precision: 81%, recall: 85%) in identifying hate speech. The study is significant in that it publicly shares a unique codeswitched dataset for hate speech that is valuable for comparative studies. Secondly, it provides a methodology for building a novel PDC feature set to identify nuanced forms of hate speech, camouflaged in codeswitched data, which conventional methods could not adequately identify.

Psychosocial Features for Identifying Hate Speech in Social Media Text

This study uses natural language processing to identify hate speech in social media codeswitched text. It trains nine models and tests their predictiveness in recognizing hate speech in a 50k human-annotated dataset. The article proposes a novel hierarchical approach that leverages Latent Dirichlet Analysis to develop topic models that assist build a high-level Psychosocial feature set we call PDC. PDC organizes words into word families, which helps capture codeswitching during preprocessing for supervised learning models. Informed by the duplex theory of hate, the PDC features are based on a hate speech annotation framework. Frequency-based models employing the PDC feature on tweets from the 2012 and 2017 Kenyan presidential elections yielded an f-score of 83 percent (precision: 81 percent, recall: 85 percent) in recognizing hate speech. The study is notable because it publicly exposes a rich codeswitched dataset for comparative studies. Second, it describes how to create a novel PDC feature set to detect subtle types of hate speech hidden in codeswitched data that previous approaches could not detect.

Automatic Speech Classifier for Mild Cognitive Impairment and Early Dementia

The World Health Organization estimates that 50 million people are currently living with dementia worldwide and this figure will almost triple by 2050. Current pharmacological treatments are only symptomatic, and drugs or other therapies are ineffective in slowing down or curing the neurodegenerative process at the basis of dementia. Therefore, early detection of cognitive decline is of the utmost importance to respond significantly and deliver preventive interventions. Recently, the researchers showed that speech alterations might be one of the earliest signs of cognitive defect, observable well in advance before other cognitive deficits become manifest. In this article, we propose a full automated method able to classify the audio file of the subjects according to the progress level of the pathology. In particular, we trained a specific type of artificial neural network, called autoencoder, using the visual representation of the audio signal of the subjects, that is, the spectrogram. Moreover, we used a data augmentation approach to overcome the problem of the large amount of annotated data usually required during the training phase, which represents one of the most major obstacles in deep learning. We evaluated the proposed method using a dataset of 288 audio files from 96 subjects: 48 healthy controls and 48 cognitively impaired participants. The proposed method obtained good classification results compared to the state-of-the-art neuropsychological screening tests and, with an accuracy of 90.57%, outperformed the methods based on manual transcription and annotation of speech.

Multilingual speech annotation of landmarks and other acoustic cues to distinctive features

A speech annotation system developed for English that identifies landmarks and other acoustic cues to distinctive features (Stevens, 2002; Huilgol et al., 2019) has been extended to Spanish and Korean. This process includes retrieving the (allo)phone sequence for the words of a target utterance from a standard lexicon, converting this sequence into an underlying phoneme sequence, and then generating citation-form landmark and other acoustic cue sequences from the phoneme sequence and its features. This standard set is presented to labelers who examine the waveform of the target utterance and mark which of the predicted cues have been realized, or deleted, and where unpredicted cues have been inserted. For each language, we draw from a superset of acoustic cues that function in all three languages, but each language also makes use of an additional set of acoustic cues specific to that language.

An automatic Alzheimer’s disease classifier based on spontaneous spoken English

According to the World Health Organization, the number of people suffering from dementia worldwide will grow to 150 million by mid-century, and Alzheimer’s disease is the most common form of dementia contributing to 60%–70% of cases. The problem is compounded by the fact that current pharmacologic treatments are only symptomatic, and therapies are ineffective in slow down or cure the degenerative process. An automatic and standardize classifier for Alzheimer’s disease is thereby extremely important to rapidly respond and deliver as preventive as possible interventions. Speech alterations might be one of the earliest signs of cognitive defect and, recently, the researchers showed that they can be observable well in advance other cognitive deficits become manifest. In this paper, we propose a full automated method able to classify the spontaneous spoken production of the subjects. In particular, we trained an artificial neural network using the spectrogram of the audio signal, which is the visual representation of the speech of the subject. Moreover, to overcome the problem of the large amount of annotated data usually required for training deep learning models, we used a specific data augmentation approach that avoids distorting the original samples. We evaluated the proposed method using the English Pitt Corpus from DementiaBank. The used dataset consists of 180 subjects: 43 healthy controls and 137 Alzheimer’s disease patients. The proposed method outperformed the other approaches in the literature based on manual and semi-automatic transcription and annotation of speech, improving the classification capability by 5.93%, and obtained good classification results compared to the state-of-the-art neuropsychological screening tests (i.e., the Mini-Mental State Examination and the Activities of Daily Living portion of the Blessed Dementia Rating Scale) exhibiting an accuracy of 93.30% and an F1 score of 88.50%.

Spatial position constraint for unsupervised learning of speech representations.

The success of supervised learning techniques for automatic speech processing does not always extend to problems with limited annotated speech. Unsupervised representation learning aims at utilizing unlabelled data to learn a transformation that makes speech easily distinguishable for classification tasks, whereby deep auto-encoder variants have been most successful in finding such representations. This paper proposes a novel mechanism to incorporate geometric position of speech samples within the global structure of an unlabelled feature set. Regression to the geometric position is also added as an additional constraint for the representation learning auto-encoder. The representation learnt by the proposed model has been evaluated over a supervised classification task for limited vocabulary keyword spotting, with the proposed representation outperforming the commonly used cepstral features by about 9% in terms of classification accuracy, despite using a limited amount of labels during supervision. Furthermore, a small keyword dataset has been collected for Kadazan, an indigenous, low-resourced Southeast Asian language. Analysis for the Kadazan dataset also confirms the superiority of the proposed representation for limited annotation. The results are significant as they confirm that the proposed method can learn unsupervised speech representations effectively for classification tasks with scarce labelled data.

Non-native acoustic modeling for mispronunciation verification based on language adversarial representation learning

Non-native mispronunciation verification is designed to provide feedback to guide language learners to correct their pronunciation errors in their further learning and it plays an important role in the computer-aided pronunciation training (CAPT) system. Most existing approaches focus on establishing the acoustic model directly using non-native corpus thus they are suffering the data sparsity problem due to time-consuming non-native speech data collection and annotation tasks. In this work, to address this problem, we propose a pre-trained approach to utilize the speech data of two native languages (the learner’s native and target languages) for non-native mispronunciation verification. We set up an unsupervised model to extract knowledge from a large scale of unlabeled raw speech of the target language by making predictions about future observations in the speech signal, then the model is trained with language adversarial training using the learner’s native language to align the feature distribution of two languages by confusing a language discriminator. In addition, sinc filter is incorporated at the first convolutional layer to capture the formant-like feature. Formant is relevant to the place and manner of articulation. Therefore, it is useful not only for pronunciation error detection but also for providing instructive feedback. Then the pre-trained model serves as the feature extractor in the downstream mispronunciation verification task. Through the experiments on the Japanese part of the BLCU inter-Chinese speech corpus, the experimental results demonstrate that for the non-native phone recognition and mispronunciation verification tasks (1) the knowledge learned from two native languages speech with the proposed unsupervised approach is useful for these two tasks (2) our proposed language adversarial representation learning is effective to improve the performance (3) formant-like feature can be incorporated by introducing sinc filter to further improve the performance of mispronunciation verification.

A studyforrest extension, an annotation of spoken language in the German dubbed movie "Forrest Gump" and its audio-description.

Here we present an annotation of speech in the audio-visual movie "Forrest Gump" and its audio-description for a visually impaired audience, as an addition to a large public functional brain imaging dataset ( studyforrest.org). The annotation provides information about the exact timing of each of the more than 2500 spoken sentences, 16,000 words (including 202 non-speech vocalizations), 66,000 phonemes, and their corresponding speaker. Additionally, for every word, we provide lemmatization, a simple part-of-speech-tagging (15 grammatical categories), a detailed part-of-speech tagging (43 grammatical categories), syntactic dependencies, and a semantic analysis based on word embedding which represents each word in a 300-dimensional semantic space. To validate the dataset's quality, we build a model of hemodynamic brain activity based on information drawn from the annotation. Results suggest that the annotation's content and quality enable independent researchers to create models of brain activity correlating with a variety of linguistic aspects under conditions of near-real-life complexity.

Hybrid-task learning for robust automatic speech recognition

In order to properly train an automatic speech recognition system, speech with its annotated transcriptions is most often required. The amount of real annotated data recorded in noisy and reverberant conditions is extremely limited, especially compared to the amount of data than can be simulated by adding noise to clean annotated speech. Thus, using both real and simulated data is important in order to improve robust speech recognition, as this increases the amount and diversity of training data (thanks to the simulated data) while also benefiting from a reduced mismatch between training and operation of the system (thanks to the real data). Another promising method applied to speech recognition in noisy and reverberant conditions is multi-task learning. The idea is to train one acoustic model to solve simultaneously at least two tasks that are different but related, with speech recognition being the main task. A successful auxiliary task consists of generating clean speech features using a regression loss (as a denoising auto-encoder). This auxiliary task though uses as targets clean speech, which implies that real data cannot be used. In order to tackle this problem a Hybrid-Task Learning system is proposed. This system switches frequently between multi and single-task learning depending on whether the input is real or simulated data respectively. Having a hybrid architecture allows us to benefit from both real and simulated data while using a denoising auto-encoder as auxiliary task of a multi-task setup. We show that the relative improvement brought by the proposed hybrid-task learning architecture can reach up to 4.4% compared to the traditional single-task learning approach on the CHiME4 database. We also demonstrate the benefits of the hybrid approach compared to multi-task learning or adaptation.

An open-source end-to-end ASR system for Brazilian Portuguese using DNNs built from newly assembled corpora

In this work, we present a baseline end-to-end system based on deep learning for automatic speech recognition in Brazilian Portuguese. To build such a model, we employ a speech corpus containing 158 hours of annotated speech by assembling four individual datasets, three of them publicly available, and a text corpus containing 10.2 millions of sentences. We train an acoustic model based on the DeepSpeech 2 network, with two convolutional and five bidirectional recurrent layers. By adding a newly trained 15-gram language model at the character level, we achieve a character error rate of only 10.49% and a word error rate of 25.45%, which are on a par with other works in different languages using a similar amount of training data.

Real-time transcription, keyword spotting, archival and retrieval for telugu TV news using ASR

The main objective of this paper is to describe the system developed for transcription, keyword spotting and alerting, archival, and retrieval for broadcasted Telugu TV news. Real-time automatic speech recognition system is developed for searching the given keywords from the transcribed text of the broadcasted Telugu TV news. The user can set the required keywords and the system will continuously track the audio and alert in real time the user as and when the keywords appeared in the audio. The system will transcribe and index every word with time. As and when a user wanted to retrieve the data with a particular word, it will search the transcribed text in the archives and play the corresponding audio, video along with scrolling of transcribed text of audio. The system is developed fully with GUI for giving the keywords either through microphone or keyboard by transliteration in Telugu. The ASR system is developed using the annotated speech corpus of 65 h duration with two hundred thousand unique words. The acoustic models are developed using subspace Gaussian mixture model. The language modeling is developed with Witten Bell and Kneser–Ney smoothing techniques

Dialogue analysis: a case study on the New Testament

There has been much research on the nature of dialogues in the Bible. While the research literature abounds with qualitative analyses on these dialogues, they are rarely corroborated on statistics from the entire text. In this article, we leverage a corpus of annotated direct speech in the New Testament, as well as recent advances in automatic speaker and listener identification, to present a quantitative study on dialogue structure in the Gospels. The contributions of this article are three-fold. First, we quantify a variety of features that are widely used in characterizing dialogue structure—including dialogue length, turn length, and the initiation and conclusion of a dialogue—and show how they distinguish between different Gospels. Second, we compare our statistics with qualitative comments in the New Testament research literature, and extend them to cover the entirety of the Gospels. Most significantly, we gauge the feasibility of applying our approach to other literary works, by measuring the amount of errors that would be introduced by automatically identified dialogues, speakers and listeners.

Developing a Thai emotional speech corpus from Lakorn (EMOLA)

Advances in emotional speech recognition and synthesis essentially rely on the availability of annotated emotional speech corpora. As a low resource language, the Thai language critically lacks corpora of emotional speech, although a few corpora have been constructed for speech recognition and synthesis. This paper presents the design of a Thai emotional speech corpus (namely EMOLA), its construction and annotation process, and its analysis. In the corpus design, four basic types with twelve subtypes of emotions are defined with consideration of the Pleasure-Arousal-Dominance emotional state model. To construct the corpus, a series of Thai dramas (1397 min) were selected and its video clips of approximately 868 min were annotated. As a result, 8987 transcriptions (of conversation turns) were derived in total, with each transcription tagged as one basic type and a few subtypes. Finally, an analysis was conducted to describe the characteristics of this corpus in three sets of statistics: collection-level, annotator-oriented and actor-oriented statistics.

Assessing automatic VOT annotation using unimpaired and impaired speech

Investigating speech processes often involves analysing data gathered by phonetically annotating speech recordings. Yet, the manual annotation of speech can often be resource intensive—requiring substantial time and labour to complete. Recent advances in automatic annotation methods offer a way to reduce these annotation costs by replacing manual annotation. For researchers and clinicians, the viability of automatic methods depends whether one can draw similar conclusions about speech processes from automatically annotated speech as one would from manually annotated speech. Here, we evaluate how well one automatic annotation tool, AutoVOT, can approximate manual annotation. We do so by comparing analyses of automatically and manually annotated speech in two studies. We find that, with some caveats, we are able to draw the same conclusions about speech processes under both annotation methods. The findings suggest that automatic methods may be a viable way to reduce phonetic annotation costs in the right circumstances. We end with some guidelines on if and how well AutoVOT may be able to replace manual annotation in other data sets.

Re-ranking spoken term detection with acoustic exemplars of keywords

Spoken term detection (STD) systems rank hypothesized detections by scores, which indicate how confident a hypothesized detection is a true instance of the keyword. Many STD systems rely on automatic speech recognition (ASR) to transcribe the speech content into the lattice representation. In such STD systems, the detection scores are usually estimated as the posterior probabilities of the keyword in the decoding lattices. Such scores may be inaccurate, e.g. due to the imperfect modeling of speech and noise. To improve the ranking of hypothesized detections, we propose to directly utilize the acoustic similarity scores between the speech signal of hypothesized detections and that of the keyword exemplars. A keyword exemplar is a true instance of the keyword obtained from an annotated speech corpus. When no exemplar is available, we propose to synthesize exemplars from the annotated speech corpus. Given the acoustic similarity between the hypothesized detections and keyword exemplars, two re-ranking methods are proposed, i.e. re-ranking by score fusion and re-ranking by similarity graph. Experimental results on the NIST OpenKWS14 and OpenKWS15 datasets show that the proposed re-ranking framework significantly outperforms the ranking based only on ASR confidence scores and also other re-ranking methods without using keyword exemplars.

Learning Prosodic Stress from Data in Neural Network based Text-to-Speech Synthesis

Naturalness is one of the most important aspects of synthesized speech, and state-of-the-art parametric speech synthesizers require training on large quantities of annotated speech data to be able to convey prosodic elements such as pitch accent and phrase boundary tone. The most frequently used framework for prosodic annotation of speech in American English is Tones and Break Indices – ToBI, which has also been adapted for use in a number of other languages. This paper presents certain deficiencies of ToBI when applied in synthesis of speech in American English, which are related to the absence of tags specifically intended to mark differences in the level of prosodic stress (emphasis) related to a particular sentence constituent. The research presented in the paper proposes the introduction of a set of tags intended for explicit modeling of the degree of prosodic stress. Namely, a certain sentence constituent can be particularly emphasized, when it is the intended focus of the utterance, or it can be de-emphasized, as is commonly the case with phrases reporting direct speech or with comment clauses. Through several listening tests it has been shown that learning such prosodic events from data has distinct advantages over approaches attempting to exploit the existing ToBI tags to convey the degree of emphasis in synthesized speech. Namely, speech synthesized by a neural network trained on data tagged for the level of prosodic stress appears more natural, and the listeners are more successful in locating the sentence constituent carrying prosodic stress.

Illocutionary Force and Perlocutionary Effect Recognition in the (Semi-)Automated Pragmatic Annotation of ELF Spoken Discourse in Migration Domains

The adoption of English as a Lingua Franca (henceforth ELF; cf. Seidlhofer 2001; Guido 2008) for mutual understanding by and among interactants with different lingua-cultural backgrounds has become a widespread and well-documented phenomenon and scholars in the field have focused their attention on numerous aspects of its use (business transactions; attitudes of speakers; Facebook interactions; pedagogy; sociolinguistics; discourse cohesion; migration domains; respectively Cogo et al. 2011; Centonze 2013, 2015c, 2016b, forthcoming). The present contribution is part of a research project (Centonze 2017, forthcoming) that aims at introducing a novel approach to spoken discourse in ELF which combines corpus pragmatics (Aijmer and Ruhlemann 2015; cf. Aijmer 2002, 2013; Anderson and Corbett 2009) - a relatively new research area in the field of language and discourse studies - with the most recent techniques of quantitative/qualitative analysis and corpus annotation by means of semi-automated software. More specifically, the present research study will focus on the annotation of speech acts from an ELF perspective and on the analysis of speech acts in their frequencies and collocations in a study corpus by means of DART (the Dialogue Annotation Research Tool v 1.1, Weisser 2015), i.e. a research tool which, among other things, includes the pragmatic annotation of spoken discourse. The interest in speech act annotation in the present PhD research project takes as a bedrock for analysis Austin’s (1962) and Searle’s (1975) speech act taxonomy and a critical review of it, which highlights both strengths and weaknesses of such an approach with a consequent need for the implementation of a new categorization system which would be fit for the purposes of the analysis of speech acts in a corpus in English as a Lingua Franca in migration encounters; hence, in the present research project speech acts are not perceived as pre-constructed categories on which to base further studies on discourse but rather undergo a process of resemanticization, re formulation as well as re-contextualization within the framework of a corpusdriven approach applied to ELF. The corpus which is being taken into consideration will be referred to as the ELF MiDo Corpus (English as a Lingua Franca in Migration DOmains corpus) and consists of over 50,000 words of conversation between asylum seekers and intercultural mediators in symmetrical contexts. All the different corpus interviews and interactions are transcribed according to a basic mark-up format (.XML) which proved to be a necessary condition for the whole corpus to be properly scanned for analysis through the DART interface. Corpora subsections are also differentiated according to the typology of speech event and interactant (e.g. informal conversation; meeting; intercultural mediator vs. asylum seeker) - analysis of speech acts is also carried out accordingly.

Altruistic Crowdsourcing for Arabic Speech Corpus Annotation

Crowdsourcing is an emerging collaborative approach that can be used for effective annotations of linguistic resources. There are many crowdsourcing genres: paid-for, games with a purpose, or altruistic (volunteer-based) approaches. In this paper, we investigate the use of altruistic crowdsourcing for speech corpora annotation by narrating our experience of validating a semi-automatic task for dialect annotation of Kalam’DZ, a corpus dedicated to Arabic Algerian dialectal varieties. We start by describing the whole process of designing altruistic crowdsourcing project. Using the unpaid crowdcrafting platform, we have performed experiments on a sample of 10% of Kalam’DZ corpus, totaling more than 10 h with 1012 speakers. The evaluation of this crowdsourcing job is ensured through a comparison with a gold standard annotation done by experts which affirms a high level of inter-annotation agreements of 81%. Our results confirm that altruistic crowdsourcing is an effective approach for speech dialect annotation. In addition, we present a set of best practices for altruistic crowdsourcing for corpus annotations.

Semi-Semantic Annotation: A guideline for the URDU.KON-TB treebank POS annotation

This work elaborates the semi-semantic part of speech annotation guidelines for the URDU.KON-TB treebank: an annotated corpus. A hierarchical annotation scheme was designed to label the part of speech and then applied on the corpus. This raw corpus was collected from the Urdu Wikipedia and the Jang newspaper and then annotated with the proposed semi-semantic part of speech labels. The corpus contains text of local & international news, social stories, sports, culture, finance, religion, traveling, etc. This exercise finally contributed a part of speech annotation to the URDU.KON-TB treebank. Twenty-two main part of speech categories are divided into subcategories, which conclude the morphological, and semantical information encoded in it. This article reports the annotation guidelines in major; however, it also briefs the development of the URDU.KON-TB treebank, which includes the raw corpus collection, designing & employment of annotation scheme and finally, its statistical evaluation and results. The guidelines presented as follows, will be useful for linguistic community to annotate the sentences not only for the national language Urdu but for the other indigenous languages like Punjab, Sindhi, Pashto, etc., as well.

A phonetic study of Nanai vowels: Using automated post-transcriptional processing techniques

Recent years have witnessed a remarkable growth in Altaic field linguistics, with recognizable achievements especially in language description and archiving. However, relatively less effort has been made to analyze the collected materials from the general phonetic perspectives. In this paper, the authors demonstrate how the archived language data can be analyzed using automated tools for speech annotation and phonetic analysis and thus with drastically reduced time and effort. Using the published IPA-based transcriptions of the lexical items (Ko & Yurn 2011), we ran Prosodylabaligner (Gorman et al. 2011) for the automatic segmentation and labelling of the Nanai recordings collected by the Altaic Society of Korea. After this post-processing, the acoustic characteristics of vowels were automatically measured using scripts written for Praat (Boersma & Weenink 2014). The acoustic analysis of Nanai vowels shows that, by and large, the automated process produces relatively reliable results with its effectiveness evidenced in terms of time and effort. We believe that Altaic linguistics will be highly benefited from these practical techniques with the improvement of accuracy.