Lexical Modalities Research Articles

Interpersonal synchrony across vocal and lexical modalities in interactions involving children with autism spectrum disorder.

Quantifying behavioral synchrony can inform clinical diagnosis, long-term monitoring, and individualised interventions in neuro-developmental disorders characterized by deficit in communication and social interaction, such as autism spectrum disorder. In this work, three different objective measures of interpersonal synchrony are evaluated across vocal and linguistic communication modalities. For vocal prosodic and spectral features, dynamic time warping distance and squared cosine distance of (feature-wise) complexity are used, and for lexical features, word mover's distance is applied to capture behavioral synchrony. It is shown that these interpersonal vocal and linguistic synchrony measures capture complementary information that helps in characterizing overall behavioral patterns.

Characterizing the semantic and form-based similarity spaces of the mental lexicon by means of the multi-arrangement method.

Collecting human similarity judgments is instrumental to measuring and modeling neurocognitive representations (e.g., through representational similarity analysis) and has been made more efficient by the multi-arrangement task. While this task has been tested for collecting semantic similarity judgments, it is unclear whether it also lends itself to phonological and orthographic similarity judgments of words. We have extended the task to include these lexical modalities and compared the results between modalities and against computational models. We find that similarity judgments can be collected for all three modalities, although word forms were considered more difficult to sort and resulted in less consistent inter- and intra-rater agreement than semantics. For all three modalities we can construct stable group-level representational similarity matrices. However, these do not capture significant idiosyncratic similarity information unique to each participant. We discuss the potential underlying causes for differences between modalities and their effect on the application of the multi-arrangement task.

Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities.

The affective state of people changes in the course of conversations and these changes are expressed externally in a variety of channels, including facial expressions, voice, and spoken words. Recent advances in automatic sensing of affect, through cues in individual modalities, have been remarkable; yet emotion recognition is far from a solved problem. Recently, researchers have turned their attention to the problem of multimodal affect sensing in the hope that combining different information sources would provide great improvements. However, reported results fall short of the expectations, indicating only modest benefits and occasionally even degradation in performance. We develop temporal Bayesian fusion for continuous real-value estimation of valence, arousal, power, and expectancy dimensions of affect by combining video, audio, and lexical modalities. Our approach provides substantial gains in recognition performance compared to previous work. This is achieved by the use of a powerful temporal prediction model as prior in Bayesian fusion as well as by incorporating uncertainties about the unimodal predictions. The temporal prediction model makes use of time correlations on the affect sequences and employs estimated temporal biases to control the affect estimations at the beginning of conversations. In contrast to other recent methods for combination of modalities our model is simpler, since it does not model relationships between modalities and involves only a few interpretable parameters to be estimated from the training data.

Linguistic Markers and Emotional Intensity

The purpose of this experiment is to explore possible relationships between the intensity of emotions and the lexical modalities for expressing those emotions. In this experiment, 60 Hebrew-speaking subjects were asked to watch four short films that aroused emotion. Two of the films gave rise to different degrees of happiness, and two produced sadness. At the end of each film, subjects were asked to report on their emotions. This experiment was based on the supposition that there is a relationship between the various lexical modalities used by the subjects when writing about their emotions and the intensity of those emotions. The lexical modalities examined included intensifiers, the use of emotion words, repetitions, the use of first person singular, the use of metaphors, and the use of exclamations, among others. This experiment supported the research hypothesis, as it confirmed that significant differences existed between the lexical modalities found in texts written with lower emotional intensity (for both happiness and sadness) and those written with greater emotional intensity.

Dyslexic and Category-Specific Aphasic Impairments in a Self-Organizing Feature Map Model of the Lexicon

DISLEX is an artificial neural network model of the mental lexicon. It was built to test computationally whether the lexicon could consist of separate feature maps for the different lexical modalities and the lexical semantics, connected with ordered pathways. In the model, the orthographic, phonological, and semantic feature maps and the associations between them are formed in an unsupervised process, based on cooccurrence of the lexical symbol and its meaning. After the model is organized, various damage to the lexical system can be simulated, resulting in dyslexic and category-specific aphasic impairments similar to those observed in human patients.