Linguistic Cues Research Articles

Multimodal Assessment of Schizophrenia Symptom Severity From Linguistic, Acoustic and Visual Cues.

Assessing the condition of every schizophrenia patient correctly normally requires lengthy and frequent interviews with professionally trained doctors. To alleviate the time and manual burden on those mental health professionals, this paper proposes a multimodal assessment model that predicts the severity level of each symptom defined in Scale for the Assessment of Thought, Language, and Communication (TLC) and Positive and Negative Syndrome Scale (PANSS) based on the patient's linguistic, acoustic, and visual behavior. The proposed deep-learning model consists of a multimodal fusion framework and four unimodal transformer-based backbone networks. The second-stage pre-training is introduced to make each off-the-shelf pre-trained model learn the pattern of schizophrenia data more effectively. It learns to extract the desired features from the view of its modality. Next, the pre-trained parameters are frozen, and the light-weight trainable unimodal modules are inserted and fine-tuned to keep the number of parameters low while maintaining the superb performance simultaneously. Finally, the four adapted unimodal modules are fused into a final multimodal assessment model through the proposed multimodal fusion framework. For the purpose of validation, we train and evaluate the proposed model on schizophrenia patients recruited from National Taiwan University Hospital, whose performance achieves 0.534/0.685 in MAE/MSE, outperforming the related works in the literature. Through the experimental results and ablation studies, as well as the comparison with other related multimodal assessment works, our approach not only demonstrates the superiority of our performance but also the effectiveness of our approach to extract and integrate information from multiple modalities.

A Spatial Hierarchical Reasoning Network for Remote Sensing Visual Question Answering

For visual question answering on remote sensing (RSVQA), current methods scarcely consider geospatial objects typically with large-scale differences and positional sensitive properties. Besides, modeling and reasoning the relationships between entities have rarely been explored, which leads to one-sided and inaccurate answer predictions. In this article, a novel method called spatial hierarchical reasoning network (SHRNet) is proposed, which endows a remote sensing (RS) visual question answering (VQA) system with enhanced visual–spatial reasoning capability. Specifically, a hash-based spatial multiscale visual representation module is first designed to encode multiscale visual features embedded with spatial positional information. Then, spatial hierarchical reasoning is conducted to learn the high-order inner group object relations across multiple scales under the guidance of linguistic cues. Finally, a visual-question (VQ) interaction module is employed to learn an effective image–text joint embedding for the final answer predicting. Experimental results on three public RS VQA datasets confirm the effectiveness and superiority of our model SHRNet.

Retracted: Semantic Role Labeling Integrated with Multilevel Linguistic Cues and Bi‐LSTM‐CRF

Retracted: Semantic Role Labeling Integrated with Multilevel Linguistic Cues and Bi‐LSTM‐CRF

Linguistic marker of schizotypy: a study on nominal reference

Semantic impairments are considered linguistic identifiers of schizotypal personality disorder. Particularly, failures in building reference of nominal expressions have been said to distinguish schizophrenia, high-schizotypy and control groups, and these failures reflect difficulties in using linguistic contextual cues. Our experimental study investigated covariances between schizotypal traits, in a nonclinical population, and interpretations of definite singular nominal expressions as referring either to kinds of objects (generic reading) or to specific objects (specific reading). The experiment was conducted in Brazilian Portuguese, where specific reading is the strong/default meaning of definite singular nominals, while generic reading is the weak meaning, being dependent on contextual information. Results indicate associations between interpretation of definite singular nominals and two schizotypal traits: constricted affect (a negative trait) and unusual perceptual experiences (a positive trait). Speakers in general had preference for assigning specific interpretations to definite singular nominals. Those with higher measurements of constricted affect and lower traits of unusual perceptual experiences, however, showed significant higher preference for this reading with an additive effect in discourse-contexts containing cues for specificity. This suggests that schizotypal traits might manifest themselves within language as a tendency to hold onto strong meaning.

Effective Prediction and Important Counseling Experience for Perceived Helpfulness of Social Question and Answering-Based Online Counseling: An Explainable Machine Learning Model.

The social question answering based online counseling (SQA-OC) is easy access for people seeking professional mental health information and service, has become the crucial pre-consultation and application stage toward online counseling. However, there is a lack of efforts to evaluate and explain the counselors' service quality in such an asynchronous online questioning and answering (QA) format efficiently. This study applied the notion of perceived helpfulness as a public's perception of counselors' service quality in SQA-OC, used computational linguistic and explainable machine learning (XML) methods suited for large-scale QA discourse analysis to build an predictive model, explored how various sources and types of linguistic cues [i.e., Linguistic Inquiry and Word Count (LIWC), topic consistency, linguistic style similarity, emotional similarity] contributed to the perceived helpfulness. Results show that linguistic cues from counselees, counselors, and synchrony between them are important predictors, the linguistic cues and XML can effectively predict and explain the perceived usefulness of SQA-OC, and support operational decision-making for counselors. Five helpful counseling experiences including linguistic styles of "talkative", "empathy", "thoughtful", "concise with distance", and "friendliness and confident" were identified in the SQA-OC. The paper proposed a method to evaluate the perceived helpfulness of SQA-OC service automatically, effectively, and explainable, shedding light on the understanding of the SQA-OC service outcome and the design of a better mechanism for SQA-OC systems.

Maltreated and non-maltreated children's truthful and dishonest reports: Linguistic and syntactic differences.

Adults are typically poor judges of the veracity of statements, requiring the need for alternative methods for detecting lies. One alternative method to human lie-detectors is using computer-based linguistic analysis which may present a more reliable method for detecting dishonesty. Moreover, while previous research has examined linguistic differences between typically developing children's and adults' truthful and dishonest reports, no study to date has examined whether maltreated children exhibit different linguistic cues to dishonesty. Thus, the current study examined maltreated and nonmaltreated children's linguistic and syntactic cues to children's truthful and dishonest reports. Nine- to 12-year-olds, half of whom were maltreated, played a computer game with a confederate: half of the children experienced a transgression (i.e., playing a forbidden game and crashing the computer) and were coached to conceal it, and half of the children experienced no transgression (i.e., simply played a computer game). All children were then interviewed about the event. The current study utilized automated linguistic and syntactic analysis software to compare children's truthful reports (no transgression occurred) with dishonest reports. Results indicated that maltreated and non-maltreated children did not differ in their indicators of dishonesty. Dishonest reporters used more first-person plural pronouns and cognitive mechanism terms and had less syntactically complex reports compared to truthful reporters. Finally, first-personal plural pronouns, cognitive mechanism terms, and syntactic complexity accurately classified (74.2%) the veracity of children's reports. The current findings present a new indicator of dishonesty (syntactic complexity) and suggest that indicators from typically developing populations may apply to maltreated children when coaching occurred.

Language-emotion interaction modulates selective attention to a speaker’s eyes and mouth: evidence from Chinese-English bilinguals

ABSTRACT Speech listeners focus on a speaker's face to acquire different information in social communication. Fixation on the mouth associates with language processing and attention to the eyes is mainly driven by social/emotional cues. Here, we investigated how selective attention to the eyes and mouth would vary with language-emotion interaction during speech processing when unbalanced Chinese-English bilinguals [Chinese, the first language (L1); English, a later-learned language (LX) of upper-intermediate proficiency] viewed a speaker narrating stories of different valence in their L1 and LX. Results showed that when listening to negative stories in Chinese relative to English, bilinguals attended to the eyes more, indicating that LX emotionality is weaker than that in L1, reducing gaze to the eyes where human adults primarily seek for emotional cues. In Chinese, compared with neutral speech, both positive and negative stories increased mouth attention and this effect was not observed in English, suggesting a particularly important role of the mouth in processing emotional information in speech and confirming emotionality reduction in LX. Our study reveals for the first time how linguistic and emotional cues integrated in speech alters attention allocation on speakers’ face to facilitate both speech comprehension and emotional perception in social communication.

Beyond a Lush Life: Representation and Performativity at America's [Only] Queer Jazz Festival

Beyond a Lush Life: Representation and Performativity at America's [Only] Queer Jazz Festival

Graded bilingual effects on attentional network function in Chinese high school students

AbstractEffective communication in multilingual environments requires bilinguals to constantly monitor linguistic cues. It is hypothesized that the constant need to monitor may result in improved attention. However, previous investigations have reported mixed, often null results, with positive findings attributed to non-linguistic variables. To address these issues, we investigated whether higher levels of bilingualism were associated with improved attentional function in a sample of culturally and socioeconomically homogenous Mandarin–English speaking bilingual adolescents. Participants completed the Attention Network Task to assess attentional network function. Data were analyzed using linear mixed-effects models in order to account for nontrivial differences. Mixed results provide partial support for domain general cognitive benefits associated with higher levels of bilingualism. Both improved and reduced performance differed based on the specific dimension of bilingualism and the attentional network assessed. Findings support the conclusion that separable dimensions of bilingual language experience assert different influences on attentional network function.

The development of spoken word recognition in informative and uninformative sentence contexts

Although there is ample evidence documenting the development of spoken word recognition from infancy to adolescence, it is still unclear how development of word-level processing interacts with higher-level sentence processing, such as the use of lexical–semantic cues, to facilitate word recognition. We investigated how the ability to use an informative verb (e.g., draws) to predict an upcoming word (picture) and suppress competition from similar-sounding words (pickle) develops throughout the school-age years. Eye movements of children from two age groups (5–6 years and 9–10 years) were recorded while the children heard a sentence with an informative or neutral verb (The brother draws/gets the small picture) in which the final word matched one of a set of four pictures, one of which was a cohort competitor (pickle). Both groups demonstrated use of the informative verb to more quickly access the target word and suppress cohort competition. Although the age groups showed similar ability to use semantic context to facilitate processing, the older children demonstrated faster lexical access and more robust cohort suppression in both informative and uninformative contexts. This suggests that development of word-level processing facilitates access of top-down linguistic cues that support more efficient spoken language processing. Whereas developmental differences in the use of semantic context to facilitate lexical access were not explained by vocabulary knowledge, differences in the ability to suppress cohort competition were explained by vocabulary. This suggests a potential role for vocabulary knowledge in the resolution of lexical competition and perhaps the influence of lexical competition dynamics on vocabulary development.

Linguistic structure modulates attention in reading: Evidence from negative concord in Italian

ABSTRACT We report the reading performance of an Italian speaker with egocentric Neglect Dyslexia on sentences with Negative Concord structures, which contain a linguistic cue to the presence of a preceding negative marker and compare it to sentences with no such cue. As predicted, the frequency of reading the whole sentence, including the initial negative marker non, was higher in Negative Concord structures than in sentences which also started with non, but crucially, lacked the medially positioned linguistic cue to the presence of non. These data support the claim that the presence of linguistic cues to sentence structure modulates attention during reading in Neglect Dyslexia.

Linguistic and non-linguistic cues in motion event endpoint description: The selection between English to and towards

The current study aims to explore the factors that could affect people’s description of a motion event endpoint. The study conducted by Liao, Dijkstra, and Zwaan (2021, Language and Cognition, 13[2], 161–190) found that two non-linguistic factors (i.e., the actor’s goal and the interlocutor’s social status) affect people’s choice between two Dutch directional prepositions (i.e., naar and richting) during event description tasks. The current study aims to extend these findings by examining the choice between a similar pair of directional prepositions in English (i.e., to and towards). Moreover, we aim to study whether grammatical aspect (i.e., the English simple present and the English progressive aspect) affects the sensitivity to the two non-linguistic factors and consequently also affects how people describe a motion event endpoint. In Experiment 1, we used the English simple present for all sentence stimuli (e.g., he walks (?) the trash bin). We found a significant effect of Interlocutor (the interlocutor’s social status) on preposition choice, but no significant effect of Intention (the actor’s goal). In Experiment 2, we replaced the English simple present with the English progressive aspect (e.g., he is walking (?) the trash bin). We found significant main effects of both Interlocutor and Intention on preposition choice. These findings extend those reported in Liao et al. (2021) Language, Cognition and Neuroscience, 35(4), 498–520 in that protagonist intention and interlocutor status were found to indeed affect motion event endpoint description. The current findings furthermore show that grammatical aspect affects people’s sensitivity to these factors, thus also affecting how a motion event endpoint is described.

Identifying psychiatric manifestations in schizophrenia and depression from audio-visual behavioural indicators through a machine-learning approach

Schizophrenia (SCZ) and depression (MDD) are two chronic mental disorders that seriously affect the quality of life of millions of people worldwide. We aim to develop machine-learning methods with objective linguistic, speech, facial, and motor behavioral cues to reliably predict the severity of psychopathology or cognitive function, and distinguish diagnosis groups. We collected and analyzed the speech, facial expressions, and body movement recordings of 228 participants (103 SCZ, 50 MDD, and 75 healthy controls) from two separate studies. We created an ensemble machine-learning pipeline and achieved a balanced accuracy of 75.3% for classifying the total score of negative symptoms, 75.6% for the composite score of cognitive deficits, and 73.6% for the total score of general psychiatric symptoms in the mixed sample containing all three diagnostic groups. The proposed system is also able to differentiate between MDD and SCZ with a balanced accuracy of 84.7% and differentiate patients with SCZ or MDD from healthy controls with a balanced accuracy of 82.3%. These results suggest that machine-learning models leveraging audio-visual characteristics can help diagnose, assess, and monitor patients with schizophrenia and depression.

Quantifying gender biases towards politicians on Reddit.

Despite attempts to increase gender parity in politics, global efforts have struggled to ensure equal female representation. This is likely tied to implicit gender biases against women in authority. In this work, we present a comprehensive study of gender biases that appear in online political discussion. To this end, we collect 10 million comments on Reddit in conversations about male and female politicians, which enables an exhaustive study of automatic gender bias detection. We address not only misogynistic language, but also other manifestations of bias, like benevolent sexism in the form of seemingly positive sentiment and dominance attributed to female politicians, or differences in descriptor attribution. Finally, we conduct a multi-faceted study of gender bias towards politicians investigating both linguistic and extra-linguistic cues. We assess 5 different types of gender bias, evaluating coverage, combinatorial, nominal, sentimental and lexical biases extant in social media language and discourse. Overall, we find that, contrary to previous research, coverage and sentiment biases suggest equal public interest in female politicians. Rather than overt hostile or benevolent sexism, the results of the nominal and lexical analyses suggest this interest is not as professional or respectful as that expressed about male politicians. Female politicians are often named by their first names and are described in relation to their body, clothing, or family; this is a treatment that is not similarly extended to men. On the now banned far-right subreddits, this disparity is greatest, though differences in gender biases still appear in the right and left-leaning subreddits. We release the curated dataset to the public for future studies.

Characteristics of Early Expressive Vocabulary in Mandarin-Speaking Children With Cochlear Implants.

The aim of this study is to evaluate whether Mandarin-speaking children with cochlear implants (CIs) demonstrated early lexical composition similar to their hearing peers who were at the same vocabulary level and the extent to which children with CIs were sensitive to linguistic and conceptual properties when developing early lexicon. Participants were 77 Mandarin-speaking children with CIs who received CIs before 30 months of age. Their expressive vocabulary was documented using the Infant Checklist of the Early Vocabulary Inventory for Mandarin Chinese 9 or 12 months after CI activation. Percent social words, common nouns, predicates (verbs, adjectives), and closed-class words in total vocabulary were computed for children at different vocabulary levels. Common nouns and verbs were further coded for their word class (noun, verb), word frequency, word length, and imageability to predict how likely a given noun or verb would be produced by children with CIs. Like children with typical hearing, social words were the most dominant category when vocabulary size in children with CIs was smaller than 20 words; common nouns became the most dominant category when the vocabulary size reached 21 words. The difference in percent common nouns and percent predicates (i.e., noun bias) was similar in children with CIs and their hearing peers. In addition, verbs, common words, monosyllabic words, and more imageable words were more likely to be produced by children with CIs than their counterparts. Mandarin children with CIs showed language-specific patterns in early lexical composition like their hearing peers. They were able to use multiple linguistic and conceptual cues when approaching early expressive vocabulary despite perceptual and processing constraints. https://doi.org/10.23641/asha.21357723.

Trimodal prediction of speaking and listening willingness to help improve turn-changing modeling.

Participants in a conversation must carefully monitor the turn-management (speaking and listening) willingness of other conversational partners and adjust their turn-changing behaviors accordingly to have smooth conversation. Many studies have focused on developing actual turn-changing (i.e., next speaker or end-of-turn) models that can predict whether turn-keeping or turn-changing will occur. Participants' verbal and non-verbal behaviors have been used as input features for predictive models. To the best of our knowledge, these studies only model the relationship between participant behavior and turn-changing. Thus, there is no model that takes into account participants' willingness to acquire a turn (turn-management willingness). In this paper, we address the challenge of building such models to predict the willingness of both speakers and listeners. Firstly, we find that dissonance exists between willingness and actual turn-changing. Secondly, we propose predictive models that are based on trimodal inputs, including acoustic, linguistic, and visual cues distilled from conversations. Additionally, we study the impact of modeling willingness to help improve the task of turn-changing prediction. To do so, we introduce a dyadic conversation corpus with annotated scores of speaker/listener turn-management willingness. Our results show that using all three modalities (i.e., acoustic, linguistic, and visual cues) of the speaker and listener is critically important for predicting turn-management willingness. Furthermore, explicitly adding willingness as a prediction task improves the performance of turn-changing prediction. Moreover, turn-management willingness prediction becomes more accurate when this joint prediction of turn-management willingness and turn-changing is performed by using multi-task learning techniques.

EEG analysis based on dynamic visual stimuli

This paper reviews best practices for experimental design and analysis for sign language research using neurophysiological methods, such as electroencephalography (EEG) and other methods with high temporal resolution, as well as identifies methodological challenges in neurophysiological research on natural sign language processing. In particular, we outline the considerations for generating linguistically and physically well-controlled stimuli accounting for 1) the layering of manual and non-manual information at different timescales, 2) possible unknown linguistic and non-linguistic visual cues that can affect processing, 3) variability across linguistic stimuli, and 4) predictive processing. Two specific concerns with regard to the analysis and interpretation of observed event related potential (ERP) effects for dynamic stimuli are discussed in detail. First, we discuss the “trigger/effect assignment problem”, which describes the difficulty of determining the time point for calculating ERPs. This issue is related to the problem of determining the onset of a critical sign (i.e., stimulus onset time), and the lack of clarity as to how the border between lexical (sign) and transitional movement (motion trajectory between individual signs) should be defined. Second, we discuss possible differences in the dynamics within signing that might influence ERP patterns and should be controlled for when creating natural sign language material for ERP studies. In addition, we outline alternative approaches to EEG data analyses for natural signing stimuli, such as the timestamping of continuous EEG with trigger markers for each potentially relevant cue in dynamic stimuli. Throughout the discussion, we present empirical evidence for the need to account for dynamic, multi-channel, and multi-timescale visual signal that characterizes sign languages in order to ensure the ecological validity of neurophysiological research in sign languages.

CHALLENGING THE ORAL-ONLY NARRATIVE: ENHANCING EARLY SIGNED INPUT FOR DEAF CHILDREN WITH HEARING PARENTS.

Learning a language is, at its core, a process of noticing patterns in the language input surrounding the learner. Although many of these language patterns are complex and difficult for adult speakers/signers to recognize, infants are able to find and learn them from the youngest age, without explicit instruction. However, this impressive feat is dependent on children's early access to ample and well-formed input that displays the regular patterns of natural language. Such input is far from guaranteed for the great majority of deaf and hard of hearing (DHH) children, leading to well-documented difficulties and delays in linguistic development. Efforts to remedy this situation have focused disproportionately on amplifying DHH children's hearing levels, often through cochlear implants, as young as possible to facilitate early access to spoken language. Given the time required for cochlear implantation, its lack of guaranteed success, and the critical importance of exposing infants to quality language input as early as possible, a bimodal bilingual approach can optimize DHH infants' chances for on-time language development by providing them with both spoken and signed language input from the start. This paper addresses the common claim that signing with DHH children renders the task of learning spoken language more difficult, leading to delays and inferior language development, compared to DHH children in oral-only environments. That viewpoint has most recently been articulated by Geers et al. (2017a), which I will discuss as a representative of the many studies promoting an oral-only approach. Contrary to their claims that signing degrades the language input available to DHH children, recent research has demonstrated that the formidable pattern-finding skills of newborn infants extends to linguistic cues in both the spoken and signed modalities, and that the additional challenge of simultaneously acquiring two languages is offset by important "bilingual advantages." Of course, securing early access to high quality signed input for DHH children from hearing families requires considerable effort, especially since most hearing parents are still novice signers. This paper closes with some suggestions for how to address this challenge through partnerships between linguistics researchers and early intervention programs to support family-centered bimodal bilingual development for DHH children.

Backers Beware: Characteristics and Detection of Fraudulent Crowdfunding Campaigns.

Crowdfunding has seen an enormous rise, becoming a new alternative funding source for emerging companies or new startups in recent years. As crowdfunding prevails, it is also under substantial risk of the occurrence of fraud. Though a growing number of articles indicate that crowdfunding scams are a new imminent threat to investors, little is known about them primarily due to the lack of measurement data collected from real scam cases. This paper fills the gap by collecting, labeling, and analyzing publicly available data of a hundred fraudulent campaigns on a crowdfunding platform. In order to find and understand distinguishing characteristics of crowdfunding scams, we propose to use a broad range of traits including project-based traits, project creator-based ones, and content-based ones such as linguistic cues and Named Entity Recognition features, etc. We then propose to use the feature selection method called Forward Stepwise Logistic Regression, through which 17 key discriminating features (including six original and hitherto unused ones) of scam campaigns are discovered. Based on the selected 17 key features, we present and discuss our findings and insights on distinguishing characteristics of crowdfunding scams, and build our scam detection model with 87.3% accuracy. We also explore the feasibility of early scam detection, building a model with 70.2% of classification accuracy right at the time of project launch. We discuss what features from which sections are more helpful for early scam detection on day 0 and thereafter.

Someone out there? A study on the social presence of anthropomorphized chatbots

Companies are increasingly employing text-based chatbots as a time and cost-efficient way to interact with customers. While companies begin to explore anthropomorphic chatbot designs by imbuing chatbots with human-like characteristics, the effectiveness of chatbot anthropomorphism remains unclear. We conducted three experiments to assess the effectiveness of chatbot anthropomorphism in customer–chatbot interactions. By equipping chatbots with human-like linguistic cues, we evoke different levels of chatbot anthropomorphism. Our results show significant positive effects of chatbot anthropomorphism on trust, purchase intention, word of mouth, and satisfaction with the shopping experience. More importantly, we identify social presence as the underlying mediating mechanism of these effects. These effects are robust and not contingent on different shopping contexts distinguished by hedonic versus utilitarian shopping motivations or the disclosure of (non-)sensitive information by customers. The present research derives managerial implications for companies that seek to effectively employ chatbots in customer interactions. Further, this study advances research on customers’ reactions towards anthropomorphized chatbots and demonstrates that social presence is a critical driver of successful customer-chatbot interactions.