Mouth Actions Research Articles

Multi-CNN based logical reasoning system for facial expression recognition on small-sample datasets

Facial expression recognition has long been an area of great interest across a wide range of fields. Deep learning is commonly employed in facial expression recognition and demonstrates excellent performance in large-sample classification tasks. However, deep learning models often encounter challenges when confronted with small-sample expression classification problems, as they struggle to extract sufficient relevant features from limited data, resulting in subpar performance. This paper presents a novel approach called the Multi-CNN Logical Reasoning System, which is based on local area recognition and logical reasoning. It initiates the process by partitioning facial expression images into two distinct components: eye action and mouth action. Subsequently, it utilizes logical reasoning based on the inherent relationship between local actions and global expressions to facilitate facial expression recognition. Throughout the reasoning process, it not only incorporates manually curated knowledge but also acquires hidden knowledge from the raw data. Experimental results conducted on two small-sample datasets derived from the KDEF and RaFD datasets demonstrate that the proposed approach exhibits faster convergence and higher prediction accuracy when compared to classical deep learning-based algorithms.

Brain activation during processing of mouth actions in patients with disorders of consciousness.

In the past 2 decades, several attempts have been made to promote a correct diagnosis and possible restorative interventions in patients suffering from disorders of consciousness. Sensory stimulation has been proved to be useful in sustaining the level of arousal/awareness and to improve behavioural responsiveness with a significant effect on oro-motor functions. Recently, action observation has been proposed as a stimulation strategy in patients with disorders of consciousness, based on neurophysiological evidence that the motor cortex can be activated not only during action execution but also when actions are merely observed in the absence of motor output, or during listening to action sounds and speech. This mechanism is provided by the activity of mirror neurons. In the present study, a group of patients with disorders of consciousness (11 males, 4 females; median age: 55 years; age range: 19-74 years) underwent task-based functional MRI in which they had, in one condition, to observe and listen to the sound of mouth actions, and in another condition, to listen to verbs with motor or abstract content. In order to verify the presence of residual activation of the mirror neuron system, the brain activations of patients were compared with that of a group of healthy individuals (seven males, eight females; median age: 33.4 years; age range: 24-65 years) performing the same tasks. The results show that brain activations were lower in patients with disorders of consciousness compared with controls, except for primary auditory areas. During the audiovisual task, 5 out of 15 patients with disorders of consciousness showed only residual activation of low-level visual and auditory areas. Activation of high-level parieto-premotor areas was present in six patients. During the listening task, three patients showed only low-level activations, and six patients activated also high-level areas. Interestingly, in both tasks, one patient with a clinical diagnosis of vegetative state showed activations of high-level areas. Region of interest analysis on blood oxygen level dependent signal change in temporal, parietal and premotor cortex revealed a significant linear relation with the level of clinical functioning, assessed with coma recovery scale-revised. We propose a classification of the patient's response based on the presence of low-level and high-level activations, combined with the patient's functional level. These findings support the use of action observation and listening as possible stimulation strategies in patients with disorders of consciousness and highlight the relevance of combined methods based on functional assessment and brain imaging to provide more detailed neuroanatomical specificity about residual activated areas at both cortical and subcortical levels.

Development of a Sign Repetition Task for Novice L2 Signers

ABSTRACT ENGLISH: There is a lack of tests available for assessing sign language proficiency among L2 learners. We have therefore developed a sign repetition test, SignRepL2, with a specific focus on the phonological features of signs. This paper describes the two phases of developing this test. In the first phase, content was developed in the form of 50 items with sentence lengths between one and three signs. Then, when a period of teaching revealed a ceiling effect in the first version, a second version was developed with 40 items varying between one and four signs. Test scores revealed increasing proficiency in Swedish Sign Language during education, and that mouth actions have a lower degree of accuracy than manual parameters.

River mouth morphodynamics and deflection over the short term: effects on spit growth and mangrove dynamics

The river mouth shows the influence of the dynamics of sediment deposition and the processes organising the deposits. The river mouths of the Guyana coast, as with other coastal systems in the Guianas, are influenced by the deposition of mud banks from the Amazon River and other rivers. This study analysed and probed the influence of the riverine and marine influences on small river mouth morphological developments. In this study, the morphological dynamics of the system were carried out from satellite images through the comparison of the inter-annual morphodynamics of the river mouths, complemented by data from hydrodynamic processes among others. This study demonstrates that the recent advancement and availability of Synthetic Aperture Radar (SAR), remotely sensed data, allow for the classification of migrating river mouth action and processes; effective monitoring of mudflats development and spit formation; and, the exhibition of river mouth transitioning and infilling. Sentinel-1 images of the Mahaica-Mahaicony river mouth in Guyana were processed and analysed using Google Earth Engine (GEE) and ArcGIS to observe the mud dynamics and its effects on deflecting the river mouth, influencing the development of spit and mangrove dynamics. The results of the analyses show that the dynamics of the mud shoal and river mouth are governed by feedback from various estuarine and hydrodynamic processes resulting from the interactions between the river and ocean. The results have not only highlighted the importance of mud infilling and sediment build-up for spit development and river-mouth deflection but the impact of the sediment morphological dynamics on the ecosystem (mangrove) associated with the river mouth.

Crosstalk in Facial EMG and Its Reduction Using ICA

There is ample evidence that electromyography (EMG) signals from the corrugator supercilii and zygomatic major muscles can provide valuable information for the assessment of subjective emotional experiences. Although previous research suggested that facial EMG data could be affected by crosstalk from adjacent facial muscles, it remains unproven whether such crosstalk occurs and, if so, how it can be reduced. To investigate this, we instructed participants (n = 29) to perform the facial actions of frowning, smiling, chewing, and speaking, in isolation and combination. During these actions, we measured facial EMG signals from the corrugator supercilii, zygomatic major, masseter, and suprahyoid muscles. We performed an independent component analysis (ICA) of the EMG data and removed crosstalk components. Speaking and chewing induced EMG activity in the masseter and suprahyoid muscles, as well as the zygomatic major muscle. The ICA-reconstructed EMG signals reduced the effects of speaking and chewing on zygomatic major activity, compared with the original signals. These data suggest that: (1) mouth actions could induce crosstalk in zygomatic major EMG signals, and (2) ICA can reduce the effects of such crosstalk.

The Case of Mouth Action in Israeli Sign Language Discourse

This article discusses the function of a particular feature of sign language—mouth action—as it is expressed in various discourse contexts. Specifically, we examine forms of mouthing and mouth gesture as they are used in signed narrative and expository texts, highlighting the signers' choices during the production of these two text types. We propose a new perspective on mouth action types, relying on insights from the domain of discourse analysis combined with functional categories proposed in previous studies on mouth actions in sign languages. Six deaf native signers of Israeli Sign Language (ISL) produced twelve texts, yielding a corpus of 2,125 mouth actions (1,112 in narrative texts and 1,013 in expository texts). Our study points to both quantitative and qualitative differences in the use of mouth actions in the two text types. The findings emphasize the importance of the two types of mouth actions as essential means of decoding the message, and they highlight the way mouth actions serve the text and are affected by it.

Naming in Two Modalities: Manual and Mouth Gestures in Hearing and Deaf Children

Studies on various sign languages have pointed out that some kind of mouth action co-occur in sign language utterances together with other body components. Although the consistency of such phenomenon has been studied in adult signing, no research has been conducted so far on the mouth actions occurring in the signing of school-aged children. In this paper, mouth actions will be described in a developmental perspective and paralleled with co-speech gestures of hearing children. In particular, mouth actions and manual gestures will be investigated in two groups of 10 bilingual Italian/Italian Sign Language (LIS) deaf and hearing children (from 6 to 14 yrs old) by using a naming task. The aim is to parallel quantitatively and qualitatively mouth actions and co-verbal gestures in order to understand their role with respect to sign and vocal naming. Furthermore, three adult signers were considered as models for understanding the typology and occurrence of mouth action in adult signing and compare it with children signing.

Sociolinguistic Variation in Mouthings in British Sign Language: A Corpus-Based Study.

Mouth activity forms a key component of all sign languages. This can be divided into mouthings, which originate from words in the ambient spoken language, and mouth gestures, which do not. This study examines the relationship between the distribution of mouthings co-occurring with verb signs in British Sign Language (BSL) and various linguistic and social factors, using the BSL Corpus. We find considerable variation between participants and a lack of homogeneity in mouth actions with particular signs. This accords with previous theories that mouthings constitute code-blending between spoken and signed languages-similar to code-switching or code-mixing in spoken languages-rather than being a phonologically or lexically compulsory part of the sign. We also find a strong association between production of plain verbs (which are body-anchored and cannot be modified spatially) and increased mouthing. In addition, we observe significant effects of region (signers from the south of the United Kingdom mouth more than those from the north), gender (women mouth more than men), and age (signers aged 16-35 years produce fewer mouthings than older participants). We find no significant effect of language background (deaf vs. hearing family). Based on these findings, we argue that the multimodal, multilingual, and simultaneous nature of code-blending in sign languages fits well within the paradigm of translanguaging. We discuss implications of this for concepts of translanguaging, code-switching, code-mixing, and related phenomena, highlighting the need to consider not just modality and linguistic codes but also sequential versus simultaneous patterning.

Eating in Synch: An investigation of parent-infant behaviour coordination during feeding interactions

During feeding, parents have been described to move their mouth as if they were eating themselves. Such matching of behaviours between parents and their infants during face-to-face interactions represents an example of behavioural synchrony. To date, however, the function of these synchronous eating-like mouth movements by the caregiver remains unexplored. To address this question, two competing hypotheses were tested: 1) the instructional hypothesis proposing that parents make eating-like mouth movements, such as opening and closing their mouth, to demonstrate to their infants what they need to do; 2) the mimicry hypothesis suggesting that parents imitate their infant’s mouth actions to enhance affiliation. To test these hypotheses, we examined the temporal dependencies between parents’ and infants’ mouth movements. We reasoned that parents’ mouth movements would occur before their infants’ if they serve an instructional purpose, but that they would happen after, if parents mimic their infants. Additionally, we expected that parents’ matching mouth movements would be more likely when their infants looked at them in both cases. To examine these hypotheses, fifteen caregivers were observed as they were feeding their six-month-old infants. Time-window sequential analysis was conducted to quantify how likely parents were to display mouth opening and closing before or after their infants did. The results revealed that parents’ mouth movements were more likely to follow infants' movements and are thus in line with the mimicry hypothesis. Interestingly, these mouth movements of parents were independent of infant’s gaze.

Gesture errors in left and right hemisphere damaged patients: A behavioural and anatomical study

ObjectiveErroneous gesture execution is at the core of motor cognition difficulties in apraxia. While a taxonomy of errors may provide important information about the nature of the disorder, classifications are currently often inconsistent. This study aims to identify the error categories which distinguish apraxic from non-apraxic patients. Method. Two groups of mixed (bucco-facial and limb) and bucco-facial apraxic patients suffering from stroke were compared to non-apraxic, left and right hemisphere damaged patients in tasks tapping the ability to perform limb and bucco-facial actions. The errors were analysed and classified into 6 categories relating to content, configuration or movement, spatial or temporal parameters and unrecognisable actions. Furthermore, an anatomical investigation (VLMS) was conducted in the whole group of left hemisphere damaged patients to investigate potential correlates of the various error categories. Results. Although all the above error typologies may be observed, the most indicative of mixed apraxia is the content-related one in all the typologies of actions (transitive and intransitive), and configuration errors in transitive ones. Configuration and content errors in mouth actions seem to be typical of bucco-facial apraxia. Spatial errors are similar in both apraxic and right brain damaged, non-apraxic patients. A lesion mapping analysis of left-brain damaged patients demonstrates that all but the spatial error category are associated with the fronto-parietal network. Moreover, content errors are also associated with fronto-insular lesions and movement errors with damage to the paracentral territory (precentral and postcentral gyri). Spatial errors are often associated to ventral frontal lesions. Conclusions. Bucco-facial and mixed apraxic patients make different types of errors in different types of actions. Not all errors are equally indicative of apraxia. In addition, the various error categories are associated with at least partially different neural correlates.

Development of Abstract Word Knowledge.

The development of children’s word knowledge is an important testing ground for the embodied account of word meaning, which proposes that word meanings are grounded in sensorimotor systems. Acquisition of abstract words, in particular, is a noted challenge for strong accounts of embodiment. We examined acquisition of abstract word meanings, using data on development of vocabulary knowledge from early school to University ages. We tested two specific proposals for how abstract words are learned: the affective embodiment account, that emotional experience is key to learning abstract word meanings, and the learning through language proposal, that abstract words are acquired through language experience. We found support for the affective embodiment account: word valence, interoception, and mouth action all facilitated abstract word acquisition more than concrete word acquisition. We tested the learning through language proposal by investigating whether words that appear in more diverse linguistic contexts are earlier acquired. Results showed that contextual diversity facilitated vocabulary acquisition, but did so for both abstract and concrete words. Our results provide evidence that emotion and sensorimotor systems are important to children’s acquisition of abstract words, but there is still considerable variance to be accounted for by other factors. We offer suggestions for future research to examine the acquisition of abstract vocabulary.

Children's exposures to boron and biocides from slime products in Asian regions.

Chemical exposure from slime toys can cause potential health effects. Although slime toys are popular in Asia, the regulation of boron, chloromethylisothiazolinone (CMIT), and methylisothiazolinone (MIT) in these toys has been implemented only in Korea. In this study, we investigated boron migration levels and CMIT and MIT concentrations in 127 slime products from Asian cities and estimated children's exposure to boron and the biocides in Korea. Slime boron migration levels were analyzed by inductively coupled plasma optical emission spectrometry (ICP-OES), according to Part 3 of the European Standard on the safety of toys (EN 71-3:2013). CMIT and MIT concentrations were analyzed using liquid chromatography tandem mass spectrometry (LC-MS/MS). Exposure doses of boron, CMIT, and MIT were calculated using the exposure pattern of children slime users in Korea. Average boron migration levels of the slime products in Seoul, Bangkok, Hong Kong, Taipei, and Yokohama were 691.9, 851.1, 806.6, 509.0, and 956.2 mg/kg, respectively. Of the 127 slime products tested, 70.1% exceeded the EU boron migration limit of 300 mg/kg for toys made with sticky material. Boron migration levels in slime products in Korea did not differ significantly by manufacturing date, although the Korean boron regulations were enacted in January 2019. CMIT and MIT were detected from 51.2% and 36.2% of all products, respectively, and tended to be detected simultaneously. Exposure doses of boron, CMIT, and MIT were highest among infants. Oral exposure was the most important for boron exposure. It is necessary to manage boron level in slime products and minimize exposure from hand to mouth action in infants.

Like Hand, Like Mouth: On the Role of Gesture-Linked Mouth Actions in the Evolution of Language

A number of language evolution researchers have argued that while language as we now know it is a predominately vocal affair, early language plausibly made extensive use of gesture. Relatedly, these same researchers often claim that while modern language in general uses arbitrary symbols, it is very likely that early language made extensive use of iconicity. Anyone accepting an account of early language along these lines must therefore explain how language shifted over time from a heavily gestural and iconic communication system to a predominately vocal and highly arbitrary one. This article looks at an intriguing theory advanced by Woll as to how this shift was made. We find the spirit of the theory plausible but take issue with its details. Specifically, we identify a serious tension at the core of the theory. We then go on to explore how that tension might be plausibly resolved. The result is a blueprint for a much-improved version of the theory, in our view.

Mouthings in Swedish Sign Language : An exploratory study

This paper deals with the non-manual mouth actions of Swedish Sign Language, Svenskt teckensprak (STS). Based on data from the Swedish Sign Language Corpus and the Swedish Sign Language as L2 Corpu ...

Use and acquisition of mouth actions in L2 sign language learners

Abstract This article deals with L2 acquisition of a sign language, examining in particular the use and acquisition of non-manual mouth actions performed by L2 learners of Swedish Sign Language. Based on longitudinal data from an L2 learner corpus, we describe the distribution, frequency, and spreading patterns of mouth actions in sixteen L2 learners at two time points. The data are compared with nine signers of an L1 control group. The results reveal some differences in the use of mouth actions between the groups. The results are specifically related to the category of mouthing borrowed from spoken Swedish. L2 signers show an increased use of mouthing compared to L1 signers. Conversely, L1 signers exhibit an increased use of reduced mouthing compared with L2 signers. We also observe an increase of adverbial mouth gestures within the L2 group. The results are discussed in relation to previous findings, and within the framework of cross-linguistic influence.

The articulatory in-out effect resists oral motor interference.

People prefer words with inward directed consonantal patterns (e.g., MENIKA) compared to outward patterns (KENIMA), because inward (outward) articulation movements resemble positive (negative) mouth actions such as swallowing (spitting). This effect might rely on covert articulation simulations, or subvocalizations, since it occurs also under silent reading. We tested to what degree these underlying articulation simulations are disturbed by oral motor interference. In 3 experiments (total N = 465) we interfered with these articulation simulations by employing concurrent oral exercises that induce oral motor noise while judging inward and outward words (chewing gum, Experiment 1; executing meaningless tongue movements, Experiment 2; concurrent verbalizations, Experiment 3). Across several word stimulus types, the articulatory in-out effect was not modulated by these tasks. This finding introduces a theoretically interesting case, because in contrast to many previous demonstrations regarding other motor-preference effects, the covert simulations in this effect are not susceptible to selective motor interference. (PsycINFO Database Record

Reaching and grasping with the tongue: Shared motor planning between hand actions and articulatory gestures.

Research has shown connections between articulatory mouth actions and manual actions. This study investigates whether forward-backward hand movements could be associated with vowel production processes that programme tongue fronting/backing, lip rounding/spreading (Experiment 1), and/or consonant production processes that programme tongue tip and tongue dorsum actions (Experiment 2). The participants had to perform either forward or backward hand movement and simultaneously pronounce different vowels or consonants. The results revealed a response benefit, measured in vocal and manual reaction times, when the responses consisted of front vowels and forward hand movements. Conversely, back vowels were associated with backward hand movements. Articulation of rounded versus unrounded vowels or coronal versus dorsal consonants did not produce the effect. In contrast, when the manual responses of forward-backward hand movements were replaced by precision and power grip responses, the coronal consonants [t] and [r] were associated with the precision grip, whereas the dorsal consonant [k] was associated with the power grip. We propose that the movements of the tongue body, operating mainly for vowel production, share the directional action planning processes with the hand movements. Conversely, the tongue articulators related to tongue tip and dorsum movements, operating mainly for consonant production, share the action planning processes with the precision and power grip, respectively.

Action-matching biases in monkeys (Sapajus spp.) in a stimulus-response compatibility task: Evaluating experience-dependent malleability.

Stimulus-response (S-R) compatibility effects occur when observing certain stimuli facilitate the performance of a related response and interfere with performing an incompatible or different response. Using stimulus-response action pairings, this phenomenon has been used to study imitation effects in humans, and here we use a similar procedure to examine imitative biases in nonhuman primates. Eight capuchin monkeys (Sapajus spp.) were trained to perform hand and mouth actions in a stimulus-response compatibility task. Monkeys rewarded for performing a compatible action (i.e., using their hand or mouth to perform an action after observing an experimenter use the same effector) performed significantly better than those rewarded for incompatible actions (i.e., performing an action after observing an experimenter use the other effector), suggesting an initial bias for imitative action over an incompatible S-R pairing. After a predetermined number of trials, reward contingencies were reversed; that is, monkeys initially rewarded for compatible responses were now rewarded for incompatible responses, and vice versa. In this 2nd training stage, no difference in performance was identified between monkeys rewarded for compatible or incompatible actions, suggesting any imitative biases were now absent. In a 2nd experiment, 2 monkeys learned both compatible and incompatible reward contingencies in a series of learning reversals. Overall, no difference in performance ability could be attributed to the type of rule (compatible-incompatible) being rewarded. Together, these results suggest that monkeys exhibit a weak bias toward action copying, which (in line with findings from humans) can largely be eliminated through counterimitative experience. (PsycINFO Database Record

Vocalizations Produced by Bottlenose Dolphins (Tursiops truncatus) during Mouth Actions in Aggressive and Non-aggressive Contexts

Dolphins exchange information with conspecifics using different types of vocalizations that are often associated with specific behaviors. The present study used simultaneous acoustic and video recordings of captive bottlenose dolphins (Tursiops truncatus) in Honduras to describe potential correlations between the type of mouthing behavior (open mouth, mouthing, bite) and associated vocalizations (whistle, whistle-squawk, chirp, moan, burst-pulse ‘A’, burst-pulse ‘B’, clicks). Literature on sound-behavior relationships among odontocetes with the noted infrequent systematic analyses of mouth actions highlights the need for this investigation. The influence of aggression on vocalization use was also addressed. From this observational study a series of general expectations on the interaction of sound, mouth action, and aggressive context are presented. Mouth actions are associated with vocalizations more often than not, and results suggest an overall flexible association of vocalizations during mouth actions, with the production of various sounds altered during aggressive contexts. There is an apparent distinction of frequency-modulated sounds with mouthing, suggesting that frequency parameters are an important characteristic of information exchange during mouth actions, and further that mouth actions are individually distinct behaviors. A dichotomy of burst pulse sounds in association with aggressive and non-aggressive contexts also introduces the need to analyze pulsed sounds according to inter-pulse interval. This paper proposes that there may be an interactive function to the use of vocalizations during mouth actions that is not yet understood.

The prominence of spoken language elements in a sign language

AbstractCode-blendsin sign languages consist of simultaneously articulated manual signs and spoken language words. These “mouthings” (typically silent articulations) have been observed for many different sign languages. The present study aims to investigate the extent of such bimodal code-mixing in sign languages by investigating the frequency of mouthings produced by deaf users of Sign Language of the Netherlands (NGT), their co-occurrence with pointing signs, and whether any differences can be explained by sociolinguistic variables such as regional origin and age of the signer. We investigated over 10,000 mouth actions from 70 signers, and found that the mouth and the hands are equally active during signing. Moreover, around 80 % of all mouth actions are mouthings, while the remaining 20 % are unrelated to Dutch. We found frequency differences between individual signers and a small effect for level of education, but not for other sociolinguistic variables. Our results provide genuine evidence that mouthings form an inextricable component of signed interaction. Rather than displaying effects of competition between languages or spoken language suppression, NGT signers demonstrate the potential of the visual modality to conjoin parallel information streams.