Hierarchical Temporal Structure Research Articles

Hierarchical Temporal Structuring of Speech: A Multiscale, Multimodal Framework to Inform the Assessment and Management of Neuromotor Speech Disorder.

Hierarchical temporal structuring of speech is the key to multiscale linguistic information transfer toward effective communication. This study investigated and linked the hierarchical temporal cues of the kinematic and acoustic modalities of natural, unscripted speech in neurologically healthy and impaired speakers. Thirteen individuals with amyotrophic lateral sclerosis (ALS) and 10 age-matched healthy controls performed a story-telling task. The hierarchical temporal structure of the speech stimulus was measured by (a) 26 articulatory-kinematic features characterizing the depth, phase synchronization, and coherence of temporal modulation of the tongue tip, tongue body, lower lip, and jaw, at three hierarchically nested timescales corresponding to prosodic stress, syllables, and onset-rime/phonemes, and (b) 25 acoustic features characterizing the parallel aspects of temporal modulation of five critical-spectral-band envelopes. All features were compared between groups. For each aspect of temporal modulation, the contributions of all articulatory features to the parallel acoustic features were evaluated by group. Generally consistent disease impacts were identified on the articulatory and acoustic features, manifested by reduced modulation depths of most articulators and critical-spectral-band envelopes, primarily at the timescales of syllables and onset-rime/phonemes. For healthy speakers, the strongest articulatory-acoustic relationships were found for (a) jaw and lip, in modulating stress timing, and (b) tongue tip, in modulating the timing relation between onset-rime/phonemes and syllables. For speakers with ALS, the tongue body, tongue tip, and jaw all showed the greatest contributions to modulating syllable timing. The observed disease impacts likely reflect reduced entrainment of speech motor activities to finer-grained linguistic events, presumably due to the dynamic constraints of the neuromuscular system. To accommodate these restrictions, speakers with ALS appear to use their residual articulatory motor capacities to accentuate and convey the perceptually most salient temporal cues underpinned by the syllable-centric parsing mechanism. This adaptive strategy has potential implications in managing neuromotor speech disorders.

Multiscale approaches to music and the embodied brain

A wide body of research is currently being devoted to investigating the multiscale processes across the brain and body, and the nature of their interactions. The purpose of this paper is to supplement these analyses of brain and body dynamics by providing a comprehensive account of the multiscale organisations also found in music, and ways in which these systems interact. We proceed in identifying scaling laws as a signature for multiscale features of a system and make the methodological choice of distinguishing 1) scale free structure from 2) scale free dynamics. We follow these distinctions in demonstrating how specifically i.) hierarchical temporal structures, ii.) long-range temporal correlations, and iii.) musical information as scale free structures relate to activity in the brain and body at various temporal or spatial scales during music perception and performance. Further, this is paralleled by fixing our focus to a limited choice of scale free dynamics, namely, iv.) resonance and entrainment, v.) power law distributions, and vi.) 1/f scaling. Though the ubiquitous scaling relations across musical activities, performance and music itself touch on the theme of universality, we outline how differing theoretical proposals concerning the relation between brain, body and musical environment arise in such contexts. Using these examples, we identify their overlap and divergence, and propose future directions for inquiry along these lines.

Child language and autism diagnosis impact hierarchical temporal structure of parent-child vocal interactions in early childhood.

Timing is critical to successful social interactions. The temporal structure of dyadic vocal interactions emerges from the rhythm, timing, and frequency of each individuals' vocalizations and reflects how the dyad dynamically organizes and adapts during an interaction. This study investigated the temporal structure of vocal interactions longitudinally in parent-child dyads of typically developing (TD) infants (n=49; 9-18 months; 48% male) and toddlers with ASD (n=23; 27.2 ± 5.0months; 91.3% male) to identify how developing language and social skills impact the temporal dynamics of the interaction. Acoustic hierarchical temporal structure (HTS), a measure of the nested clustering of acoustic events across multiple timescales, was measured in free play interactions using Allan Factor. HTS reflects a signal's temporal complexity and variability, with greater HTS indicating reduced flexibility of the dyadic system. Child expressive language significantly predicted HTS (ß=-0.2) longitudinally across TD infants, with greater dyadic HTS associated with lower child language skills. ASD dyads exhibited greater HTS (i.e., more rigid temporal structure) than nonverbal matched (d=0.41) and expressive language matched TD dyads (d=0.28). Increased HTS in ASD dyads occurred at timescales >1s, suggesting greater structuring of pragmatic aspects of interaction. Results provide a new window into how language development and social reciprocity serve as constraints to shape parent-child interaction dynamics and showcase a novel automated approach to characterizing vocal interactions across multiple timescales during early childhood.

Violation of rhythmic expectancies can elicit late frontal gamma activity nested in theta oscillations.

Rhythm processing involves building expectations according to the hierarchical temporal structure of auditory events. Although rhythm processing has been addressed in the context of predictive coding, the properties of the oscillatory response in different cortical areas are still not clear. We explored the oscillatory properties of the neural response to rhythmic incongruence and the cross‐frequency coupling between multiple frequencies to further investigate the mechanisms underlying rhythm perception. We designed an experiment to investigate the neural response to rhythmic deviations in which the tone either arrived earlier than expected or the tone in the same metrical position was omitted. These two manipulations modulate the rhythmic structure differently, with the former creating a larger violation of the general structure of the musical stimulus than the latter. Both deviations resulted in an MMN response, whereas only the rhythmic deviant resulted in a subsequent P3a. Rhythmic deviants due to the early occurrence of a tone, but not omission deviants, seemed to elicit a late high gamma response (60–80 Hz) at the end of the P3a over the left frontal region, which, interestingly, correlated with the P3a amplitude over the same region and was also nested in theta oscillations. The timing of the elicited high‐frequency gamma oscillations related to rhythmic deviation suggests that it might be related to the update of the predictive neural model, corresponding to the temporal structure of the events in higher‐level cortical areas.

Detecting Dynamic States of Temporal Networks Using Connection Series Tensors

Many temporal networks exhibit multiple system states, such as weekday and weekend patterns in social contact networks. The detection of such distinct states in temporal network data has recently been studied as it helps reveal underlying dynamical processes. A commonly used method is network aggregation over a time window, which aggregates a subsequence of multiple network snapshots into one static network. This method, however, necessarily discards temporal dynamics within the time window. Here we propose a new method for detecting dynamic states in temporal networks using connection series (i.e., time series of connection status) between nodes. Our method consists of the construction of connection series tensors over nonoverlapping time windows, similarity measurement between these tensors, and community detection in the similarity network of those time windows. Experiments with empirical temporal network data demonstrated that our method outperformed the conventional approach using simple network aggregation in revealing interpretable system states. In addition, our method allows users to analyze hierarchical temporal structures and to uncover dynamic states at different spatial/temporal resolutions.

3D interaction techniques for musical expression

As Virtual Reality headsets become accessible, more and more artistic applications are developed, including immersive musical instruments. 3D interaction techniques designed in the 3D User Interfaces research community, such as navigation, selection and manipulation techniques, open numerous opportunities for musical control. For example, navigation techniques such as teleportation, free walking/flying and path-planning enable different ways of accessing musical scores, scenes of spatialised sound sources or even parameter spaces. Manipulation techniques provide novel gestures and metaphors, e.g. for drawing or sculpting sound entities. Finally, 3D selection techniques facilitate the interaction with complex visual structures which can represent hierarchical temporal structures, audio graphs, scores or parameter spaces. However, existing devices and techniques were developed mainly with a focus on efficiency, i.e. minimising error rate and task completion times. They were therefore not designed with the specifics of musical interaction in mind. In this paper, we review existing 3D interaction techniques and examine how they can be used for musical control, including the possibilities they open for instrument designers. We then propose a number of research directions to adapt and extend 3DUIs for musical expression

Real-time observation of electronic, vibrational, and rotational dynamics in nitric oxide with attosecond soft x-ray pulses at 400 eV

Photoinduced quantum dynamics in molecules have hierarchical temporal structures with different energy scales that are associated with electron and nuclear motions. Femtosecond-to-attosecond transient absorption spectroscopy (TAS) using high-harmonic generation (HHG) with a photon energy below 300 eV has been a powerful tool to observe such electron and nuclear dynamics in a table-top manner. However, comprehensive measurements of the electronic, vibrational, and rotational molecular dynamics have not yet been achieved. Here, we demonstrate HHG-based TAS at the nitrogen K-edge (400 eV) for the first time, and observe all the electronic, vibrational, and rotational degrees of freedom in a nitric oxide molecule at attosecond to sub-picosecond time scales. This method of employing core-to-valence transitions offers an all-optical approach to reveal complete molecular dynamics in photochemical reactions with element and electronic state specificity.

Rhythm in speech and animal vocalizations: a cross-species perspective.

Why does human speech have rhythm? As we cannot travel back in time to witness how speech developed its rhythmic properties and why humans have the cognitive skills to process them, we rely on alternative methods to find out. One powerful tool is the comparative approach: studying the presence or absence of cognitive/behavioral traits in other species to determine which traits are shared between species and which are recent human inventions. Vocalizations of many species exhibit temporal structure, but little is known about how these rhythmic structures evolved, are perceived and produced, their biological and developmental bases, and communicative functions. We review the literature on rhythm in speech and animal vocalizations as a first step toward understanding similarities and differences across species. We extend this review to quantitative techniques that are useful for computing rhythmic structure in acoustic sequences and hence facilitate cross‐species research. We report links between vocal perception and motor coordination and the differentiation of rhythm based on hierarchical temporal structure. While still far from a complete cross‐species perspective of speech rhythm, our review puts some pieces of the puzzle together.

Neural entrainment is associated with subjective groove and complexity for performed but not mechanical musical rhythms

Both movement and neural activity in humans can be entrained by the regularities of an external stimulus, such as the beat of musical rhythms. Neural entrainment to auditory rhythms supports temporal perception, and is enhanced by selective attention and by hierarchical temporal structure imposed on rhythms. However, it is not known how neural entrainment to rhythms is related to the subjective experience of groove (the desire to move along with music or rhythm), the perception of a regular beat, the perception of complexity, and the experience of pleasure. In two experiments, we used musical rhythms (from Steve Reich’s Clapping Music) to investigate whether rhythms that are performed by humans (with naturally variable timing) and rhythms that are mechanical (with precise timing), elicit differences in (1) neural entrainment, as measured by inter-trial phase coherence, and (2) subjective ratings of the complexity, preference, groove, and beat strength of rhythms. We also combined results from the two experiments to investigate relationships between neural entrainment and subjective perception of musical rhythms. We found that mechanical rhythms elicited a greater degree of neural entrainment than performed rhythms, likely due to the greater temporal precision in the stimulus, and the two types only elicited different ratings for some individual rhythms. Neural entrainment to performed rhythms, but not to mechanical ones, correlated with subjective desire to move and subjective complexity. These data, therefore, suggest multiple interacting influences on neural entrainment to rhythms, from low-level stimulus properties to high-level cognition and perception.

Learning, Probability and Logic: Toward a Unified Approach for Content-Based Music Information Retrieval

Within the last fifteen years, the field of Music Information Retrieval (MIR) has made tremendous progress in the development of algorithms for organizing and analyzing the ever-increasing large and varied amount of music and music-related data available digitally. However, the development of content-based methods to enable or improve multimedia retrieval still remains a central challenge. In this perspective paper, we critically look at the problem of automatic chord estimation from audio recordings as a case study of content-based algorithms, and point out several bottlenecks in current approaches: expressiveness and flexibility are obtained to the expense of robustness and vice-versa; available multimodal sources of information are little exploited; modeling multi-faceted and strongly interrelated musical information is limited with current architectures; models are typically restricted to short-term analysis that does not account for the hierarchical temporal structure of musical signals. Dealing with music data requires the ability to handle both uncertainty and complex relational structure at multiple levels of representation. Traditional approaches have generally treated these two aspects separately, probability and learning being the standard way to represent uncertainty in knowledge, while logical representation being the standard way to represent knowledge and complex relational information. We advocate that the identified hurdles of current approaches could be overcome by recent developments in the area of Statistical Relational Artificial Intelligence (StarAI) that unifies probability, logic and (deep) learning. We show that existing approaches used in MIR find powerful extensions and unifications in StarAI, and we explain why we think it is time to consider the new perspectives offered by this promising research field.

A prominence-based account of temporal discourse structure

In this article, we develop an account of temporal discourse structure that is composed of three different levels and based on the notion of prominence. The levels of temporal structuring in discourse are as follows:(i)the level of the linear and hierarchical temporal structure, which is constituted by time points and time spans as well as their interrelations,(ii)the level of perspective and perspective taking, and(iii)the level of foregrounding and backgrounding and text segmentation into story lines.Although these levels have already been identified by the research literature, they have neither been explicated in a systematic way while taking prominence into consideration nor brought together in a complete descriptive account. The unifying ingredient of our account is the principle of prominence, according to which the elements of a set of equals compete as to their prominence status, with one of them standing out with respect to the others at a certain point of the unfolding discourse. We show the crucial role of prominence in the structuring of discourse on each of the three levels.Our prominence-based account highlights the hierarchical structuring of discourse. This allows us to consider all important features of temporal discourse structure in one integrated account: tense choice, aspect, adverbial expressions and connectors, discourse relations and episode structure.

Influence of metrical structure on learning of positional regularities in movement sequences.

Sequential stimuli are usually perceived to have hierarchical temporal structures. However, some of these structures are only investigated in one type of sequence, regardless of the existing evidence, showing the domain-generality of the representation of these structures. Here, we assess whether the hierarchical representation of regularly segmented action sequences resembles the perceived metrical patterns that organize the representation of events hierarchically in temporally regular sequences. In all our experiments, we presented the participants with sequences of human movements and tested the perception of metrical pattern by segmenting the movement streams into temporally equal groups containing four movements. In Experiment 1, we found that a movement sequence with temporally equal groupings improves the learning of positional regularities inherent within each group of movements. To further clarify the degree to which this learning mechanism is affected by the perceived metrical patterns, we conducted Experiments 2a and 2b, in which the relative saliencies of the first and last positions in the movement groups, respectively, were studied. The results showed that, although in the learning of positional regularities, the rule-conforming first positions are as effective as when both first and last positions are legal, the last positions are not as influential. Based on these findings we conclude that, in grouped sequences, learning of positional regularities may be modulated by the metrical saliency patterns that are imposed by the temporal regularity of the sequential grouping pattern.

Hierarchical burst model for complex bursty dynamics.

Temporal inhomogeneities observed in various natural and social phenomena have often been characterized in terms of scaling behaviors in the autocorrelation function with a decaying exponent γ, the interevent time distribution with a power-law exponent α, and the burst size distributions. Here the interevent time is defined as a time interval between two consecutive events in the event sequence, and the burst size denotes the number of events in a bursty train detected for a given time window. To understand such temporal scaling behaviors implying a hierarchical temporal structure, we devise a hierarchical burst model by assuming that each observed event might be a consequence of the multilevel causal or decision-making process. By studying our model analytically and numerically, we confirm the scaling relation α+γ=2, established for the uncorrelated interevent times, despite of the existence of correlations between interevent times. Such correlations between interevent times are supported by the stretched exponential burst size distributions, for which we provide an analytic argument. In addition, by imposing conditions for the ordering of events, we observe an additional feature of log-periodic behavior in the autocorrelation function. Our modeling approach for the hierarchical temporal structure can help us better understand the underlying mechanisms behind complex bursty dynamics showing temporal scaling behaviors.

Complexity matching in speech: Effects of speaking rate and naturalness

Recordings of speech exhibit nested clustering of peak amplitude events that reflects the hierarchical temporal structure of language. Previous studies have found variations in nested clustering to correspond with variations in prosody and social interaction. In the present study, we tested two specific dimensions of variation in speech hypothesized to have differing effects on hierarchical temporal structure: Speaking rate and naturalness. Rate was manipulated both algorithmically and experimentally, and naturalness was manipulated using synthesized speech, with sine wave speech as a comparison. Allan Factor analysis was used to quantify nested clustering of peak amplitude events in speech recordings as a function of timescale. For fast speech, nested clustering was found to shift into shorter timescales, whereas for synthesized speech, nested clustering was found to decrease in the longer timescales. Results are discussed in terms of complexity matching and its implications for how neural and perceptual processes might respond to changes in the hierarchical temporal structure of speech signals.

Hierarchical Multi-scale Attention Networks for action recognition

Recurrent Neural Networks (RNNs) have been widely used in natural language processing and computer vision. Amongst them, the Hierarchical Multi-scale RNN (HM-RNN), a recently proposed multi-scale hierarchical RNN, can automatically learn the hierarchical temporal structure from data. In this paper, we extend the work to solve the computer vision task of action recognition. However, in sequence-to-sequence models like RNN, it is normally very hard to discover the relationships between inputs and outputs given static inputs. As a solution, the attention mechanism can be applied to extract the relevant information from the inputs thus facilitating the modeling of the input–output relationships. Based on these considerations, we propose a novel attention network, namely Hierarchical Multi-scale Attention Network (HM-AN), by incorporating the attention mechanism into the HM-RNN and applying it to action recognition. A newly proposed gradient estimation method for stochastic neurons, namely Gumbel-softmax, is exploited to implement the temporal boundary detectors and the stochastic hard attention mechanism. To reduce the negative effect of the temperature sensitivity of the Gumbel-softmax, an adaptive temperature training method is applied to improve the system performance. The experimental results demonstrate the improved effect of HM-AN over LSTM with attention on the vision task. Through visualization of what has been learnt by the network, it can be observed that both the attention regions of the images and the hierarchical temporal structure can be captured by a HM-AN.

Hierarchical temporal structure in music, speech and animal vocalizations: jazz is like a conversation, humpbacks sing like hermit thrushes

Humans talk, sing and play music. Some species of birds and whales sing long and complex songs. All these behaviours and sounds exhibit hierarchical structure-syllables and notes are positioned within words and musical phrases, words and motives in sentences and musical phrases, and so on. We developed a new method to measure and compare hierarchical temporal structures in speech, song and music. The method identifies temporal events as peaks in the sound amplitude envelope, and quantifies event clustering across a range of timescales using Allan factor (AF) variance. AF variances were analysed and compared for over 200 different recordings from more than 16 different categories of signals, including recordings of speech in different contexts and languages, musical compositions and performances from different genres. Non-human vocalizations from two bird species and two types of marine mammals were also analysed for comparison. The resulting patterns of AF variance across timescales were distinct to each of four natural categories of complex sound: speech, popular music, classical music and complex animal vocalizations. Comparisons within and across categories indicated that nested clustering in longer timescales was more prominent when prosodic variation was greater, and when sounds came from interactions among individuals, including interactions between speakers, musicians, and even killer whales. Nested clustering also was more prominent for music compared with speech, and reflected beat structure for popular music and self-similarity across timescales for classical music. In summary, hierarchical temporal structures reflect the behavioural and social processes underlying complex vocalizations and musical performances.

Hierarchical organization in the temporal structure of infant-direct speech and song

Caregivers alter the temporal structure of their utterances when talking and singing to infants compared with adult communication. The present study tested whether temporal variability in infant-directed registers serves to emphasize the hierarchical temporal structure of speech. Fifteen German-speaking mothers sang a play song and told a story to their 6-months-old infants, or to an adult. Recordings were analyzed using a recently developed method that determines the degree of nested clustering of temporal events in speech. Events were defined as peaks in the amplitude envelope, and clusters of various sizes related to periods of acoustic speech energy at varying timescales. Infant-directed speech and song clearly showed greater event clustering compared with adult-directed registers, at multiple timescales of hundreds of milliseconds to tens of seconds. We discuss the relation of this newly discovered acoustic property to temporal variability in linguistic units and its potential implications for parent-infant communication and infants learning the hierarchical structures of speech and language.

Learning of hierarchical temporal structures facilitates the prediction of future events

Learning of hierarchical temporal structures facilitates the prediction of future events

A time-hierarchical microeconomic model of activities

The microeconomic approach to explain consumers’ behavior regarding the choice of activities, consumption of goods and use of time is extended in this paper by explicitly including the temporal dimension in the choice-making process. Recognizing that some activities, such as a job and education, involve a long-term commitment and that other activities, such as leisure and shopping, are conducted and modified in the short term, we make these differences explicit in a microeconomic framework. Thus, a hierarchical temporal structure defines the time window or frequency of adjusting the variables of activities (duration, location and consumption of goods) and the magnitude of the resources (time and money) spent. We specify and analyze a stylized microeconomic model with two time scales, the macro and micro level, concluding that preference observations at the micro level, such as transport mode choice, are strongly conditioned by the prevailing choices at the macro scale. This result has strong implications for the current theory of the value and allocation of time, as well as on the location of activities, as illustrated by numerical example.