Textual Affect Sensing for Sociable and Expressive Online Communication

This chapter addresses the tasks of recognition, interpretation and visualization of affect communicated through text messaging in virtual communication environments. In order to facilitate sensitive and expressive communication in such environments, we introduced a novel syntactic rule-based approach to affect recognition from text. Our Affect Analysis Model follows the compositionality principle, according to which emotional meaning of a sentence is determined by composing parts that correspond to lexical units or other linguistic constituent types governed by the rules of aggregation, propagation, domination, neutralization, and intensification, at various grammatical levels. The proposed rule-based approach processes each sentence in sequential stages, including symbolic cue processing, detection and transformation of abbreviations, sentence parsing, and word/phrase/sentence-level analyses. Our method is capable of processing sentences of different complexity, including simple, compound, complex (with complement and relative clauses), and complex-compound sentences. Affect in text is classified into nine emotion categories (or neutral), and, additionally, information that indicates social communicative behaviour is identified. The evaluation of the Affect Analysis Model algorithm showed promising results regarding its capability to accurately recognize affective information in text from an existing corpus of informal online conversations. The applications of the developed Affect Analysis Model in Instant Messaging system (AffectIM) and in Second Life (EmoHeart, iFeel_IM!) are described in the chapter.

In this paper, we address the task of affect recognition from text messaging. In order to sense and interpret emotional information expressed through written language, rule-based affect analysis system employing natural language processing techniques was created. Since the purpose of our work is to improve social interactivity and affective expressiveness of computer-mediated communication, we decided to tailor the system to handle style and specifics of online conversations. Proposed algorithm for affect analysis covers symbolic cue processing, detection and transformation of abbreviations, sentence parsing, and word/phrase/sentence-level analyses. To realize visual reflection of textual affective information, we have designed an avatar displaying emotions, social behaviour, and natural idle movements.

In this paper, we address the tasks of recognition and interpretation of affect communicated through text messaging in online communication environments. Specifically, we focus on Instant Messaging (IM) or blogs, where people use an informal or garbled style of writing. We introduced a novel rule-based linguistic approach for affect recognition from text. Our Affect Analysis Model (AAM) was designed to deal with not only grammatically and syntactically correct textual input, but also informal messages written in an abbreviated or expressive manner. The proposed rule-based approach processes each sentence in stages, including symbolic cue processing, detection and transformation of abbreviations, sentence parsing and word/phrase/sentence-level analyses. Our method is capable of processing sentences of different complexity, including simple, compound, complex (with complement and relative clauses) and complex–compound sentences. Affect in text is classified into nine emotion categories (or neutral). The strength of the resulting emotional state depends on vectors of emotional words, relations among them, tense of the analysed sentence and availability of first person pronouns. The evaluation of the Affect Analysis Model algorithm showed promising results regarding its capability to accurately recognize fine-grained emotions reflected in sentences from diary-like blog posts (averaged accuracy is up to 77 per cent), fairy tales (averaged accuracy is up to 70.2 per cent) and news headlines (our algorithm outperformed eight other systems on several measures).

Unlike sentiment analysis which detects positive, negative, or neutral sentences, textual affect sensing tries to detect more detailed affective or emotional states appearing in text, such as joy, sadness, anger, fear, disgust, surprise and much more. The authors describe here their following two approaches for textual affect sensing: The first one detects nine emotions using a set of rules implemented on the basis of a linguistic compositionality principle for textual affect interpretation. This process includes symbolic cue processing, detection and transformation of abbreviations, sentence parsing, and word/phrase/sentence-level analyses. The second one challenged to recognize 22 emotion types defined in the OCC (Ortony, Clore & Collins) emotion model, which is the most comprehensive emotion model and employs several cognitive variables. In this research, we have shown how these cognitive variables of the emotion model can be computed from linguistic components in text. These two approaches have exploited detailed level analyses of text in two different ways more than ever towards textual affect sensing. Applications towards affective communication are also outlined, including affective instant messaging, affective chat in 3D virtual world, affective haptic interaction, and online news classification relying on affect.

In addition to semantic content, affect conveyed by text plays an important role for rich and friendly communication. This is particularly true in human communication. In recent days, the percentage of human-computer and computer-mediated communications is increasing in our life. In this situation, a computer is expected to understand the affects or emotions included in text. We have been working on this problem, i.e., textual affect sensing, for some years. As a related topic, textual sentiment analysis has been studied, where positive and negative sentences are typically extracted for Web opinion mining with respect to a specific issue or product. While the distinction between affect sensing and sentiment analysis is not necessarily clear in this field, I call here sentiment analysis when a sentence is classified into positive, negative or neutral one. Unlike this sort of sentiment analysis, our textual affect sensing detects more detailed affective or emotional states appearing in text, such as happy, sad, anger, fear, disgust, surprise and much more. We basically have developed the following two such models or systems so far: (A) The first one detects nine emotions using a set of rules implemented on the basis of a compositionality principle proposed for textual affect interpretation. This process includes symbolic cue processing, detection and transformation of abbreviations, sentence parsing, and word/phrase/sentence-level analyses. (B) The second one challenged to recognize 22 emotion types defined in the OCC (Ortony, Clore & Collins) emotion model, which is the most comprehensive emotion model and employs several cognitive variables including one relating to valenced reactions of events or agents. In this research, we have shown how these cognitive variables of the emotion model can be computed from linguistic components in text.

The average American college student text messages constantly throughout the day. According to a Boston area Verizon retailer, it is not uncommon for young adults (an age range he did not define) to send 8,000 text messages a month. Texting has become an increasingly more important form of communication in our culture, and our country’s colleges are the hotbeds of linguistic activity and change within this particular medium. Its emergence and popularity have not gone unnoticed in the sociolinguistic community, where text messaging and online conversation have become rich new areas of linguistic data yet unexplored. Most famous in this discussion is British linguist David Crystal, whose recent research, culminating in his 2008 book Txting: The Gr8 Deb8, has incited discussion and suggested fascinating conclusions about the sophistication of this means of communication and its influence on the English language. Crystal writes a great deal about the acronyms and abbreviations within the grammar of the Short Messaging Service (SMS), but in this paper I focused on the relationship of tone and punctuation, a topic that emerged out of a conversation with a fellow student. She was struggling to compose a message to a recent acquaintance (and potential romantic interest), and could not decide on the proper end punctuation for the attitude she wanted to project. The message content, something mundane about her hometown in Maryland, wasn’t the problem; it was deciding the appropriate way to finish the message off, to apply a tone. We ran through the list of options and discussed all of their implications, trying to identify which one would best convey her overall attitude: interested and engaged, but definitely light and casual. After trying a few different combinations of haha’s and exclamation points we landed on something satisfactory and she sent it off. Reflecting on it later, I realized that we had just achieved a pretty complex linguistic act. With limited resources we managed to make sure her recipient knew how she felt; her message served as a vehicle for an emotional transfer. Text messages that are pure content are all business, just an instant communication of needs to another person who could be in any place and engaged in any activity. Without explicit markers of tone, such a message is bold and possibly disconcerting, and not at all analogous to face-to-face conversation, in which politeness is paramount. American college students therefore employ punctuation and add certain particles in order to avoid ambiguity of tone and preserve standards of politeness in text messaging.

Theories and practices in second language reading pedagogy often overlook the sentence processing description from the psycholinguistics perspective. Second language reading comprehension is easily associated with vocabulary learning or discourse strategy. Yet, such activities can lead to an unnatural way of reading such as translating vocabularies or pointing out information as required. Meanwhile the authentic way of reading should encourage a natural stream of ideas to be interpreted from sentence to sentence. As suggested by the sentence processing notion from the psycholinguistics point of view, syntax appears to be the key to effective and authentic reading as opposed to the general belief of semantic or discourse information being the primary concern. This article argues that understanding the architecture of sentence processing, with syntactic parsing at the core of the underlying mechanism, can offer insights into the second language reading pedagogy. The concepts of syntactic parsing, reanalysis, and sentence processing models are described to give the idea of how sentence processing works. Additionally, a critical review on the differences between L1 and L2 sentence processing is presented considering the recent debate on individual differences as significant indicators of nativelike L2 sentence processing. Lastly, implications for the L2 reading pedagogy and potential implementation in instructional setting are discussed.

User study on AffectIM, an avatar-based Instant Messaging system employing rule-based affect sensing from text

Our research addresses the tasks of recognition, interpretation and visualization of affect communicated through text messaging. In order to facilitate sensitive and expressive interaction in computer-mediated communication, we previously introduced a novel syntactical rule-based approach to affect recognition from text. The evaluation of the developed Affect Analysis Model showed promising results regarding its capability to accurately recognize affective information in text from an existing corpus of informal online conversations. To enrich the user's experience in online communication, make it enjoyable, exciting and fun, we implemented a web-based IM application, AffectIM, and endowed it with emotional intelligence by integrating the developed Affect Analysis Model. This paper describes the findings of a twenty-person study conducted with our AffectIM system. The results of the study indicated that automatic emotion recognition function can bring a high level of affective intelligence to the IM application.

Parental language input is a key predictor of child language achievement. Parentese is a widely used style of child-directed speech (CDS) distinguished by a higher pitch and larger pitch range. A recent parent coaching randomized control trial (Parentese-RCT) demonstrated that English-speaking US parents who were coached to use parentese with their 6- to 18-month-olds increased the frequency of its use; their children showed enhanced language outcomes at 18 months. While these results are exciting, a roadblock in scaling this intervention is the fact that parentese has to be manually identified from daylong LENA recordings. Here we demonstrate that a newly developed CDS classifier captures the effect of this Parentese-RCT without human annotation, through a new variable: Proportion of CDS relative to all adult speech (CDS-Proportion). Two daylong recordings per child (N=70) per timepoint (child age: 6, 10, 14, and 18 months) from the Parentese-RCT were re-analyzed by removing periods of sleep and estimating CDS-Proportion through the classifier. As was the case for parentese in the Parentese-RCT, CDS-Proportion was significantly enhanced in the intervention group. Moreover, the change over time in CDS-Proportion was significantly and positively correlated with infants' word use at 18 months. We emphasize that the classifier does not "recognize parentese." Likewise, CDS-Proportion is not a proxy for parentese, but rather, a variable related to parentese in complex ways. The present findings suggest a promising future for scalability of interventions using daylong recordings in combination with novel technologies. SUMMARY: Language interventions promote verbal engagement between caregivers and children; however, to be effective, these interventions need to be scalable. Language interventions have relied on manual (human) annotation to identify parental language behaviors that are linked to children's outcomes. An automated classifier for Child-Directed-Speech was applied to capture the effect of a previously published parent-coaching intervention, without human annotation. Novel technologies have the potential to enhance the scalability of parent interventions that focus on social language behaviors.

Comprehending and producing sentences is a complex endeavor requiring the coordinated activity of multiple brain regions. We examined three issues related to the brain networks underlying sentence comprehension and production in healthy individuals: First, which regions are recruited for sentence comprehension and sentence production? Second, are there differences for auditory sentence comprehension vs. visual sentence comprehension? Third, which regions are specifically recruited for the comprehension of syntactically complex sentences? Results from activation likelihood estimation (ALE) analyses (from 45 studies) implicated a sentence comprehension network occupying bilateral frontal and temporal lobe regions. Regions implicated in production (from 15 studies) overlapped with the set of regions associated with sentence comprehension in the left hemisphere, but did not include inferior frontal cortex, and did not extend to the right hemisphere. Modality differences between auditory and visual sentence comprehension were found principally in the temporal lobes. Results from the analysis of complex syntax (from 37 studies) showed engagement of left inferior frontal and posterior temporal regions, as well as the right insula. The involvement of the right hemisphere in the comprehension of these structures has potentially important implications for language treatment and recovery in individuals with agrammatic aphasia following left hemisphere brain damage.

Despite medical advances, prematurity is associated with a higher risk of neurodevelopmental problems. Although social impairments are commonly reported in preterm children, their origins are not clearly determined though they may be associated with the integrity of social cognitive skills, such as theory of mind. This study aimed to assess social cognitive, social adaptive, and social behavior functioning in children born extremely prematurely. Thirty children born between 22 and 28 weeks of gestation and 30 children born at term completed measures of social cognition (theory of mind, affect recognition) between 4:6, 5:11 years of age (mean = 5.29, standard deviation = 0.28 years). Parents completed questionnaires measuring their child’s adaptive social functioning and social behavior. Analyses of covariance controlling for cognitive, attentional, executive, sociodemographic, and perinatal characteristics were performed. Children born between 22 and 28 weeks displayed poorer theory of mind (p < 0.01) and affect recognition (p < 0.01) than term controls, and their parents reported lower adaptive social functioning (p < 0.01) and prosocial behavior (p = 0.04). Social cognitive, adaptive, and behavior functioning appear to be affected by extreme preterm birth. These findings may explain the social challenges experienced by extremely preterm children and provide potential loci for targeted interventions to optimize social functioning.

Background: Many studies report that the comprehension of sentences derived by movement of phrases is impaired in agrammatism. However, only few studies have explored the comprehension of sentences that involve a movement of the verb. In several languages, the verb can or should move to the second position of a sentence, creating VSO sentences like “Yesterday ate the girl a watermelon” from an SVO sentence. Previous studies of comprehension of verb movement either allowed the patients to use a strategy, or used grammaticality judgement tasks, which probably tap different abilities from interpretation tasks. The research was supported by a research grant from the National Institute for Psychobiology in Israel (Friedmann 2004‐5‐2b), and by the Joint German–Israeli Research Program grant (Friedmann GR‐01791). Many thanks to Ronit Szterman for her participation in the development of this experiment and to Esther Ruigendijk for her helpful comments. Aims: The present study tested the comprehension of sentences with verb movement to second position in agrammatism using a novel sentence type that prevented participants from employing strategy‐based comprehension. Comprehension was tested using sentences with verb–noun homophones and homographs. In general, the choice between the noun and the verb meaning of homophones and homographs relies on the construction of the syntactic structure of the sentence, and the syntactic role of the ambiguous word. In the current study, we used sentences in which the ambiguous word was placed at the object position, such as “Yesterday caught the bat flies in the garden” (literally transcribed into English). In order to understand whether it is a verb or a noun (whether the bat in this sentence flies, or whether it catches flies), comprehension of the relation between the moved verb and its object is required. Thus, these sentences might shed light on whether individuals with agrammatism can understand verb movement. Methods & Procedures: participants were six Hebrew‐speaking individuals with agrammatic aphasia. In Experiment 1 they paraphrased auditorily presented sentences with homophones; in Experiment 2 they read aloud and then paraphrased written sentences with heterophonic homographs. Both experiments also included, in addition to the target sentences with verb movement, matched sentences with the same homographs and homophones that did not include verb movement. Experiment 1 included 51 sentences, Experiment 2 included 48 sentences per participant. Outcomes & Results: The individuals with agrammatic aphasia failed to read and paraphrase the sentences with verb movement. They either took the object to be the verb, read the moved verb incorrectly, said they did not understand the sentence, or said that there were two parts of the sentence that did not connect. Matched sentences with the same homophones and homographs without verb movement were comprehended significantly better. Normal subjects performed correctly in all conditions. Conclusions: Not only is the comprehension of movement of phrases impaired in agrammatism, but also the comprehension of sentences derived by verb movement.

Social interactional behaviors and symbolic play competence as predictors of language development and their associations with maternal attention-directing strategies

In this article, we address the contribution of Vygotsky to contemporary knowledge of language comprehension and production including their development. We compare his concepts of ‘sense’ and ‘sense field’, or ‘inner speech field’, with state-of-the-art concepts of situation model of text and embodied comprehension. According to Vygotsky, speech comprehension comprises several sequential stages which include decoding of the external speech meaning, then its translation into senses in inner speech and, finally, identification of a thought and motive of an interlocutor. Language production is a reversed process. Therefore, in Vygotsky’s opinion, sense is the content of inner speech which mediates language production and comprehension. Vygotsky’s concepts are compatible with present understanding of the situation model of text on the deepest levels of language production and comprehension. Furthermore, his ideas have greater explanatory power, in particular, due to developmental analysis of the inner speech field. Its development implies interiorization of external forms of mental activity and divergence between optical and sense fields, resulting in emergence of the internal aspect of consciousness.