Inter-transcriber Agreement Research Articles

Clinical application of usage-based phonology: Treatment of cleft palate speech using usage-based electropalatography

Purpose To investigate whether a novel electropalatography (EPG) therapy, underpinned by usage-based phonology theory, can improve the accuracy of target speech sounds for school-aged children and adults with persistent speech sound disorder (SSD) secondary to cleft palate +/− lip. Method Six consecutively treated participants (7–27 years) with long-standing speech disorders associated with cleft palate enrolled in a multiple baseline (ABA) within-participant case series. The usage-based EPG therapy technique involved high-volume production of words. Speech was assessed on three baselines prior to therapy, during weekly therapy, at completion of therapy, and 3 months post-therapy. Percent correct of target phonemes in untreated words and continuously connected speech were assessed through acoustic phonetic transcription. Intra- and inter-transcriber agreement was determined. Result Large to medium treatment effect sizes were shown for all participants following therapy (15–33 sessions). Percentage of targets correct for untreated words improved from near 0% pre-therapy, to near 100% for most target sounds post-therapy. Generalisation of target sounds to spontaneous connected speech occurred for all participants and ranged from 78.95−100% (M = 90.66; SD = 10.14) 3 months post-therapy. Conclusion Clinically significant speech change occurred for all participants following therapy. Response to the novel therapeutic technique is encouraging and further research is indicated.

Speech feature profiles in Swedish 5-year-olds with speech sound disorder related to suspected childhood apraxia of speech or cleft palate

Purpose: To study the occurrence of speech features commonly associated with Childhood Apraxia of Speech (CAS) in Swedish children with suspected CAS (sCAS) or Speech Sound Disorder (SSD) related to Cleft Palate and/or Lip (CP ± L). Method: Thirty-four children (4.10–5.11) with SSD related to sCAS (n = 15) or repaired CP ± L (n = 19) participated. Consensus judgement of presence/absence of CAS features in single words were based on a checklist with operationalised definitions. Speech sound production measures were based on semi-narrow phonetic transcription. Intra- and inter-transcriber agreement was determined. Result: Twelve participants (ten with sCAS (67%) and two with CP ± L (11%)) shared a CAS profile of phonemic speech inconsistency for consonants and vowels and a set of four features: vowel error, voicing error, difficulty achieving initial articulatory configurations or transitionary movement gestures and stress errors. The most frequent speech difficulties in children with non-CAS CP ± L (n = 17) were consonant distortion (88%) and hypernasal resonance (76%). Prosodic impairment was rare. Conclusion: A distinct CAS speech feature profile was found for children with CAS, differing in number and distribution compared to children with CP ± L and SSD. CAS was found more frequently in CP ± L and SSD compared to reported estimates of clinical prevalence.

A system for high quality crowdsourced indigenous language transcription

In this article, a crowdsourcing method is proposed to transcribe manuscripts from the Bleek and Lloyd Collection, where non-expert volunteers transcribe pages of the handwritten text using an online tool. The digital Bleek and Lloyd Collection is a rare collection that contains artwork, notebooks and dictionaries of the indigenous people of Southern Africa. The notebooks, in particular, contain stories that encode the language, culture and beliefs of these people, handwritten in now-extinct languages with a specialized notation system. Previous attempts have been made to convert the approximately 20,000 pages of text to a machine-readable form using machine learning algorithms but, due to the complexity of the text, the recognition accuracy was low. This article presents details of the system used to enable transcription by volunteers as well as results from experiments that were conducted to determine the quality and consistency of transcriptions. The results show that volunteers are able to produce reliable transcriptions of high quality. The inter-transcriber agreement is 80 % for |Xam text and 95 % for English text. When the |Xam text transcriptions produced by the volunteers are compared with a gold standard, the volunteers achieve an average accuracy of 64.75 %, which exceeded that in previous work. Finally, the degree of transcription agreement correlates with the degree of transcription accuracy. This suggests that the quality of unseen data can be assessed based on the degree of agreement among transcribers.

The impact of speech material on speech judgement in children with and without cleft palate

The chosen method of speech assessment, including type of speech material, may affect speech judgement in children with cleft palate. To assess the effect of different speech materials on speech judgement in 5-year-old children born with or without cleft palate, as well as the reliability of materials by means of intra- and inter-transcriber agreement of consonant transcriptions. Altogether 40 children were studied, 20 born with cleft palate, 20 without. The children were audio recorded at 5 years of age. Speech materials used were: single-word naming, sentence repetition (both developed for cleft palate speech assessment), retelling of a narrative and conversational speech. The samples were phonetically transcribed and inter- and intra-transcriber agreement was calculated. Percentage correct consonants (PCC), percentage correct places (PCP), percentage correct manners (PCM), and percentage active cleft speech characteristics (CSC) were assessed. In addition, an analysis of phonological simplification processes (PSP) was performed. The PCC and CSC results were significantly more accurate in word naming than in all other speech materials in the children with cleft palate, who also achieved more accurate PCP results in word naming than in sentence repetition and conversational speech. Regarding PCM and PSP, performance was significantly more accurate in word naming than in conversational speech. Children without cleft palate did better, irrespective of the speech material. The medians of intra- and inter-transcriber agreement were good in both groups and all speech materials. The closest agreement in the cleft palate group was seen in word naming and the weakest in the retelling task. The results indicate that word naming is the most reliable speech material when the purpose is to assess the best speech performance of a child with cleft palate. If the purpose is to assess connected speech, sentence repetition is a reliable and also valid speech material, with good transcriber agreement and equally good articulation accuracy as in retelling and conversational speech. For typically developing children without a cleft palate, the chosen speech material appears not to affect speech judgement.

The role of syntactic structure in guiding prosody perception with ordinary listeners and everyday speech

The relationship between syntactic and prosodic phrase structures is investigated in the production and perception of spontaneous speech. Three hypotheses are tested: (1) syntax influences prosody production; (2) listeners' perception of prosodic boundaries is sensitive to acoustic duration; and (3) syntax directly influences boundary perception, (partly) independent of the acoustic evidence for boundaries. Data are from the Buckeye corpus of conversational speech, and the real-time prosodic transcription of those data by 97 untrained listeners. Inter-transcriber agreement codes boundary strength at word junctures, and Boundary scores are shown to be correlated with both the syntactic context and vowel duration of a word. Vowel duration is also correlated with syntactic context, but the effect of syntactic context on boundary perception is not fully explained by vowel duration. Regression analyses show that syntactic clause boundaries and vowel duration are the first and second strongest predictors of boundary perception in spontaneous speech.

Perception of prosodic boundaries in spontaneous speech with and without silent pauses.

In speech comprehension, listeners attend to variation in multiple acoustic parameters encoding prosodic structure. Given the multiplicity of acoustic cues, we ask whether prosody perception is dependent on any individual cue or whether acoustic redundancy encoding prosody supports robust prosody perception in the absence of an individual cue. The present paper reports on a study of boundary perception in spontaneous speech with and without silent pause as a boundary cue. Prior studies show that in read speech, silent pause is important for boundary perception, while in spontaneous speech, listeners can detect boundaries without pauses. Our study tests the role of pause in boundary perception with two versions of 36 short speech excerpts from the Buckeye Corpus: one with pauses intact and another with all pauses truncated to 20 ms. In real-time transcription tasks based only on auditory impression, boundary locations were marked by 74 subjects for the intact stimuli and by an additional 15 subjects for truncated excerpts. Inter-transcriber agreement was comparable across the intact and truncated conditions. Paired-sample t-tests show significantly higher rates of boundary perception for intact stimuli indicating that silent pause is an important but not necessary cue to boundary perception and cue redundancy allows for robust perception.

How do ordinary listeners perceive prosodic prominence? Syntagmatic versus paradigmatic comparison.

In comprehending speech, listeners are sensitive to the acoustic variation encoding the prosodic structures that mark phrasing and prominence. The present study, based on prosody transcriptions of untrained listeners (74 monolingual American English speakers), tests which acoustic feature or feature combinations cue prosodic prominence, and specifically, whether ordinary listeners perceive prosodic prominence based on changes in acoustic measures in the local context (syntagmatic comparison) or based on the value of an acoustic measure relative to the distribution of that measure across all instances of the specific phoneme in the listener’s experience (paradigmatic comparison). Subjects listened to 36 short excerpts (11–25 s) of spontaneous speech from the Buckeye Corpus, and marked prominent words on a transcript. After evaluating intertranscriber agreement rates, acoustic measures (duration, intensity, subband intensities, F0 max, and formants) were extracted from stressed vowels and normalized within two domains (syntagmatic/paradigmatic). The results show that all the acoustic measures are correlated with perceived prominence and combinations of the acoustic measures account for the variability in listeners’ perception of prominence. Moreover, syntagmatically normalized acoustic measures explain more of the variability, indicating that ordinary listeners perceive prosodic prominence based on local changes in acoustic measures and especially in changes in overall intensity.

Perceived prosody: Phonetic bases of prominence and boundaries.

In comprehending speech, listeners are sensitive to the prosodic features that signal the phrasing and the discourse salience of words (prominence). Findings from two experiments on prosody perception show that acoustic and articulatory kinematic properties of speech correlate with native listeners’ perception of phrasing and prominence. Subjects in this study were 114 university-age adults (74 UIUC + 40 Haskins), monolingual speakers of American English who were untrained in prosody transcription. Subjects listened to short recorded excerpts (about 20 s) from two corpora of spontaneous and read speech (Buckeye Corpus and Wisconsin Microbeam Database) and marked prominent words and the location of phrase boundaries on a transcript. Intertranscriber agreement rates across subsets of 17–40 subjects are significantly above chance based on Fleiss’ statistic, indicating that listeners’ perception of prosody is reliable, with higher agreement rates for boundary perception than for prominence. Prosody perception varies across listeners (both corpora) and across speakers (WMD, where perceived prosody varies for the same utterance produced by different speakers). Acoustic measures from stressed vowels (Buckeye: duration, intensity, F1, F2) and articulatory kinematic measures (WMD) are correlated with the perceived prosodic features of the word. [Work supported by NSF.]

Intonation units in spoken interaction: Developing transcription skills

This paper describes the transcription process and the development of transcription skills in a research project using recorded spoken interaction as its main data. The spoken data was transcribed using intonation units, and the paper traces the development of the first author's skills in identifying such intonation units. Intertranscriber checks of transcription, involving three researchers, were used to highlight ways in which the identification of intonation units could be improved. Subsequent re-transcription of the data highlighted stretches of talk that included many hesitations, false starts, and speech used to regulate ongoing spoken interaction. These features were linked to low levels of intertranscriber agreement. It is argued that the existing literature on intonation units does not address how to best deal with this quality of spontaneous spoken interaction. The paper concludes with an agenda that may be used to improve the quality of transcription in similar research projects, and to develop the transcription skills of the researchers that are responsible for transcription.

A study of manual articulatory feature-based transcription of conversational speech

This study investigates the manual labeling of speech, and in particular conversational speech, at the articulatory feature level. A detailed transcription, including subtleties such as overlapping or reduced gestures, is useful for studying the great pronunciation variability in conversational speech. This type of labeling also facilitates the testing of automatic feature classifiers, such as those used in articulatory approaches to automatic speech recognition. For this study, approximately 100 utterances drawn from the switchboard database have been transcribed using eight articulatory tiers rather than the traditional single phonetic tier. The tiers include: place and degree for up to two constrictions, nasality, glottal state, lip rounding, and vowel quality. Two transcribers have labeled this set of utterances in a multipass strategy, allowing for correction of errors. Preliminary analysis shows a high degree of intertranscriber agreement. Further analysis of the data is being performed to address a number of questions, such as: How quickly and reliably can this type of transcription be done? What are its advantages and disadvantages relative to purely phone-based transcription? What characteristics of the utterances correspond to high or low transcriber agreement? What can be learned from the data regarding articulatory phenomena such as reduction and asynchrony?

Cognitive state classification in a spoken tutorial dialogue system

This paper addresses the manual and automatic labelling, from spontaneous speech, of a particular type of user affect that we call the cognitive state in a tutorial dialogue system with students of primary and early middle school ages. Our definition of the cognitive state is based on analysis of children’s spontaneous speech, which is acquired during Wizard-of-Oz simulations of an intelligent math and physics tutor. The cognitive states of children are categorized into three classes: confidence, puzzlement, and hesitation. The manual labelling of cognitive states had an inter-transcriber agreement of kappa score 0.93. The automatic cognitive state labels are generated by classifying prosodic features, text features, and spectral features. Text features are generated from an automatic speech recognition (ASR) system; features include indicator functions of keyword classes and part-of-speech sequences. Spectral features are created based on acoustic likelihood scores of a cognitive state-dependent ASR system, in which phoneme models are adapted to utterances labelled for a particular cognitive state. The effectiveness of the proposed method has been tested on both manually and automatically transcribed speech, and the test yielded very high correctness: 96.6% for manually transcribed speech and 95.7% for automatically recognized speech. Our study shows that the proposed spectral features greatly outperformed the other types of features in the cognitive state classification experiments. Our study also shows that the spectral and prosodic features derived directly from speech signals were very robust to speech recognition errors, much more than the lexical and part-of-speech based features.

Ambiguity in Child Language: The Problem of Interobserver Reliability in Ambiguous Observation Data

In language acquisition, inter-transcriber agreement over linguistic categories assigned to recorded utterances is conceived as a measure of observer reliability. We argue that disagreement is not merely a reflection of observer errors or noisy data, but can be a reflection of the genuine ambiguity of early speech. Disagreement arises from the fact that the child is still building linguistic categories, and therefore, from the fact that the language is truly ambiguous. This ambiguity can be quantified by applying concepts from fuzzy logic, which we demonstrate in a case study. After presenting an index of agreement and a Monte-Carlo procedure for calculating the probability of chance agreement, we introduce an index of ambiguity, based on the fuzzy logic notion of degreeof-membership.

The conceptual similarity of intonational tones and its effects on intertranscriber reliability.

Tests of intertranscriber agreement in prosodically-labeled corpora have been used as an objective performance measure of reliability. Reasonably high agreements among labelers have been found, but systematic disagreements exist, indicating that some intonational patterns are more difficult for transcribers to label while others are easier. This may be due to differences in the way transcribers distinguish between tonal labels for pitch events. We developed a method to map the subjective similarity space for the categories in an intonational transcription system. From this map, we derived a conceptual tone similarity similarity index indicating the distance between tone categories. This subjective similarity index is used to predict the intertranscriber reliability. It is found that tones which are conceptually similar result in higher transcriptional disagreements while tones which are conceptually dissimilar result in lower disagreement.

Pronunciation variability in the Switchboard corpus

This paper first describes a project on manual phonetic labeling of some 2075 words from the Switchboard corpus (a 3 million word corpus of unscripted telephone conversations, recorded and orthographically transcribed by Texas Instruments, and available from the Linguistic Data Consortium). Multiple (from 10 to 40) tokens of 72 lexical items were transcribed; the transcription system used was an extension of the TIMITBET, designed to allow a narrower transcription, particularly of consonants. Intertranscriber agreement was assessed using the Oregon Graduate Institute’s metric for transcription accuracy, and comparison with their results for their English telephone corpus will be provided. A number of phonological facts will then be elucidated from the transcriptions. To facilitate this, a database of contextual information has been created for each phoneme in the dictionary forms of the words; the database includes both lexical contextual factors and the actual context in which a given token appears. The scheme is similar to that of Withgott and Chen [Computational Models of American Speech (1993)], who used such a contextual database to develop phonological rules for TIMIT. Here, the phonological rules which emerge from Switchboard will be discussed and compared with those already found in TIMIT.

Refining transcriptions: the significance of transcriber 'errors'.

ABSTRACTThis work describes the importance of the transcription process in studies of speech and language acquisition. Using data collected from a hearing child of deaf parents, the three authors derived independent transcriptions of the same speech sample and systematically compared their transcripts with each other and with the best estimate of the speaker's actual productions. The resultant transcripts were used to produce two descriptions of this child's phonological system, one based on a liberal estimate and one on a conservative estimate of the potential error in the transcripts. Discussion includes suggestions for deriving percentages of inter-transcriber agreement and the utility of such figures as a metric of transcription difficulty as well as transcriber ability.