Speech Research Articles

HumourHindiNet: Humour detection in Hindi web series using word embedding and convolutional neural network

Humour is a crucial aspect of human speech, and it is, therefore, imperative to create a system that can offer such detection. While data regarding humour in English speech is plentiful, the same cannot be said for a low-resource language like Hindi. Through this article, we introduce two multimodal datasets for humour detection in the Hindi web series. The dataset was collected from over 500 minutes of conversations amongst the characters of the Hindi web series Kota-Factory and Panchayat . Each dialogue is manually annotated as Humour or Non-Humour. Along with presenting a new Hindi language-based Humour detection dataset, we propose an improved framework for detecting humour in Hindi conversations. We start by preprocessing both datasets to obtain uniformity across the dialogues and datasets. The processed dialogues are then passed through the Skip-gram model for generating Hindi word embedding. The generated Hindi word embedding is then passed onto three convolutional neural network (CNN) architectures simultaneously, each having a different filter size for feature extraction. The extracted features are then passed through stacked Long Short-Term Memory (LSTM) layers for further processing and finally classifying the dialogues as Humour or Non-Humour. We conduct intensive experiments on both proposed Hindi datasets and evaluate several standard performance metrics. The performance of our proposed framework was also compared with several baselines and contemporary algorithms for Humour detection. The results demonstrate the effectiveness of our dataset to be used as a standard dataset for Humour detection in the Hindi web series. The proposed model yields an accuracy of 91.79 and 87.32 while an F1 score of 91.64 and 87.04 in percentage for the Kota-Factory and Panchayat datasets, respectively.

A Within-Subject Multimodal NIRS-EEG Classifier for Infant Data.

Functional Near Infrared Spectroscopy (fNIRS) and Electroencephalography (EEG) are commonly employed neuroimaging methods in developmental neuroscience. Since they offer complementary strengths and their simultaneous recording is relatively easy, combining them is highly desirable. However, to date, very few infant studies have been conducted with NIRS-EEG, partly because analyzing and interpreting multimodal data is challenging. In this work, we propose a framework to carry out a multivariate pattern analysis that uses an NIRS-EEG feature matrix, obtained by selecting EEG trials presented within larger NIRS blocks, and combining the corresponding features. Importantly, this classifier is intended to be sensitive enough to apply to individual-level, and not group-level data. We tested the classifier on NIRS-EEG data acquired from five newborn infants who were listening to human speech and monkey vocalizations. We evaluated how accurately the model classified stimuli when applied to EEG data alone, NIRS data alone, or combined NIRS-EEG data. For three out of five infants, the classifier achieved high and statistically significant accuracy when using features from the NIRS data alone, but even higher accuracy when using combined EEG and NIRS data, particularly from both hemoglobin components. For the other two infants, accuracies were lower overall, but for one of them the highest accuracy was still achieved when using combined EEG and NIRS data with both hemoglobin components. We discuss how classification based on joint NIRS-EEG data could be modified to fit the needs of different experimental paradigms and needs.

Isochrony as ancestral condition to call and song in a primate.

Animal songs differ from calls in function and structure, and have comparative and translational value, showing similarities to human music. Rhythm in music is often distributed in quantized classes of intervals known as rhythmic categories. These classes have been found in the songs of a few nonhuman species but never in their calls. Are rhythmic categories song-specific, as in human music, or can they transcend the song-call boundary? We analyze the vocal displays of one of the few mammals producing both songs and call sequences: Indri indri. We test whether rhythmic categories (a) are conserved across songs produced in different contexts, (b) exist in call sequences, and (c) differ between songs and call sequences. We show that rhythmic categories occur across vocal displays. Vocalization type and function modulate deployment of categories. We find isochrony (1:1 ratio, like the rhythm of a ticking clock) in all song types, but only advertisement songs show three rhythmic categories (1:1, 1:2, 2:1 ratios). Like songs, some call types are also isochronous. Isochrony is the backbone of most indri vocalizations, unlike human speech, where it is rare. In indri, isochrony underlies both songs and hierarchy-less call sequences and might be ancestral to both.

COMPLICATED SENTENCE AS AN INDICATOR OF A VIRTUAL SPEAKER’S INTELLECTUAL LEVEL

The paper looks at the problem of correlation of cognitive processes taking place in the person’s cognition with the syntactic organization of their speech. The speech of any individuum mirrors their thinking, and this is one of the ways they come to know their environment, acquire cognitive experience for building up their own picture of the world. In the process and due to communication, a human is shaped as an individual with a certain intellectual level of mind. The presented work is focused on syntactic peculiarities of the English-language verbal message, produced depending on the intellectual level of the virtual speaker. The topicality of the conducted scientific research consists in lingual and gnoseological need to penetrate into the cognitive-communicative essence of the processes of generating messages, and is motivated by the lack of serious scientific research of this problem. The objective of this paper consists in singling out a scientifically valid criterion for determining the intellectual level of a virtual speaker in the aspect of syntactic construction of sentences inherent to him. The starting point of this research was an assumption that there are certain typologically different structural sentence models, preferred by representatives of a certain psychological group of individuals. In our previous studies, it was established that in dialogue communication, simple sentences prevail among other structural types, while complex sentences take the third place in terms of use frequency with almost all speakers, regardless of their education level, social status, and cognitive abilities. According to our observations, a striking contrast can be seen in the native speakers’ use of complicated sentences, the grammatical status of which is not fully specified in grammar. The carried-out research specifies the concept of “complicated sentence”, and by means of comparative analysis of speech samples provided by the protagonists’ of three English novels captured at different stages of their lives, this type of sentence is determined as an indicator of the speakers’ cognitive peculiarities at the level of syntax. It is suggested that, in contrast to the surface representation, deep structures of human speech organization correlate with deep structures of human thought activity. Hence, a high intelligence level of an individuum is reflected not only at the level of adequacy of information perception, but also in the form of message representation and generation, namely, mainly in the constructions with a hidden deep structure of predication.

The Impact of Visual and Audiovisual Inputs on Voice Perception and Production: Exploring the Role of Immersive Virtual Reality in Clinical Practice

Sensory feedback is crucial for regulating pitch, loudness and other aspects of voice and speech perception and production. (1) Auditory input is traditionally considered the primary signal in speech communication. One of many examples leading to this reasonable conclusion are studies showing that deaf infants exhibit delayed articulatory precision in babbling and phonological deficits during puberty. (2,3) Other studies have found that superior auditory skills enhance pitch accuracy in voice production, revealing the relationship between auditory input and voice production. (4) Interestingly, other evidence indicates that blind children may also experience phonological deficits. (5) Additionally, incongruent audiovisual input impacts speech perception. (6) Despite these findings, the influence of visual input on individuals’ voice perception and production has largely been overlooked.

A toolkit for the dynamic study of air sacs in siamang and other elastic circular structures.

Biological structures are defined by rigid elements, such as bones, and elastic elements, like muscles and membranes. Computer vision advances have enabled automatic tracking of moving animal skeletal poses. Such developments provide insights into complex time-varying dynamics of biological motion. Conversely, the elastic soft-tissues of organisms, like the nose of elephant seals, or the buccal sac of frogs, are poorly studied and no computer vision methods have been proposed. This leaves major gaps in different areas of biology. In primatology, most critically, the function of air sacs is widely debated; many open questions on the role of air sacs in the evolution of animal communication, including human speech, remain unanswered. To support the dynamic study of soft-tissue structures, we present a toolkit for the automated tracking of semi-circular elastic structures in biological video data. The toolkit contains unsupervised computer vision tools (using Hough transform) and supervised deep learning (by adapting DeepLabCut) methodology to track inflation of laryngeal air sacs or other biological spherical objects (e.g., gular cavities). Confirming the value of elastic kinematic analysis, we show that air sac inflation correlates with acoustic markers that likely inform about body size. Finally, we present a pre-processed audiovisual-kinematic dataset of 7+ hours of closeup audiovisual recordings of siamang (Symphalangus syndactylus) singing. This toolkit (https://github.com/WimPouw/AirSacTracker) aims to revitalize the study of non-skeletal morphological structures across multiple species.

Speech-induced suppression and vocal feedback sensitivity in human cortex.

Across the animal kingdom, neural responses in the auditory cortex are suppressed during vocalization, and humans are no exception. A common hypothesis is that suppression increases sensitivity to auditory feedback, enabling the detection of vocalization errors. This hypothesis has been previously confirmed in non-human primates, however a direct link between auditory suppression and sensitivity in human speech monitoring remains elusive. To address this issue, we obtained intracranial electroencephalography (iEEG) recordings from 35 neurosurgical participants during speech production. We first characterized the detailed topography of auditory suppression, which varied across superior temporal gyrus (STG). Next, we performed a delayed auditory feedback (DAF) task to determine whether the suppressed sites were also sensitive to auditory feedback alterations. Indeed, overlapping sites showed enhanced responses to feedback, indicating sensitivity. Importantly, there was a strong correlation between the degree of auditory suppression and feedback sensitivity, suggesting suppression might be a key mechanism that underlies speech monitoring. Further, we found that when participants produced speech with simultaneous auditory feedback, posterior STG was selectively activated if participants were engaged in a DAF paradigm, suggesting that increased attentional load can modulate auditory feedback sensitivity.

Hybrid SPEAK OUT! protocol improves aerodynamic measurements and PROMs in Parkinson's disease.

SPEAK OUT! has been shown to enhance various aspects of voice such as intensity, prosody, voice quality and perception of voice. However, their impacts on expiration and communication effectiveness have not yet been evaluated. This study aimed to evaluate the effectiveness of the Hybrid SPEAK OUT! method on aerodynamic measurements and patient-reported outcome measures (PROMs) in individuals with Parkinson's disease (PD). This study included 157 participants with PD who underwent an intensive 8-week multidisciplinary treatment program. The Hybrid SPEAK OUT! method consisted of three sessions per week, including two group sessions and one individual treatment session. Voice-related aerodynamic measures included maximum phonation time (MPT), vocal cord function using the S/Z ratio, and expiratory flow measures including peakexpiratoryflow (PEF) and peak cough flow (PCF). Two PROMs were included: the Hebrew version of the voice handicap index-10 and communication effectiveness questionnaire. The results of 111 participants were analysed. MPT duration increased, PEF and PCF increased, and better scores were found in PROMs. These findings would appear to support the effectiveness of the Hybrid SPEAK OUT! methods improving function, activity and participation in individuals with PD. However, further research is needed. What is already known on the subject Prior research has demonstrated the effectiveness of behavioral therapies, including the SPEAK OUT! program, in managing speech symptoms in individuals with Parkinson's disease (PD). These therapies have shown improvements in voice intensity, fundamental frequency, voice quality, and voice perception. However, the impact on aerodynamic measures, expiratory flow, and patient-reported outcome measures has not been extensively studied. What this paper adds to existing knowledge This study adds to the existing knowledge by demonstrating that a Hybrid SPEAK OUT! approach, can lead to improvements in aerodynamic measures, including maximum phonation time (MPT), expiratory and cough peak flow. In terms of motor learning, we found that two non-speech exhalation-related tasks that were not directly trained improved following the intensive speech training protocol. Furthermore, it shows positive changes in patient-reported outcome measures, with reduced voice-related disability and improved communication efficiency. What are the potential or actual clinical implications of this work? The findings of this study have important clinical implications for the management of speech symptoms in individuals with PD. The Hybrid SPEAK OUT! program, which combines group and individual sessions, can effectively improve voice, aerodynamic measurements and patient-reported outcome measures (PROMs), ultimately enhancing the overall quality of life for patients. These findings support the effectiveness of the Hybrid SPEAK OUT! methods for improving function, activity, and participation in individuals with PD.

Delivering Script Therapy for people with aphasia in EVA Park: Two single case treatment studies

BACKGROUND: There are few applications of virtual reality (VR) in aphasia rehabilitation. EVA Park is an online VR platform developed with and for people with aphasia. Our research is testing its potential to host aphasia therapies. OBJECTIVE: Two case studies evaluated if delivery of Script Therapy in EVA Park is feasible and acceptable to participants, whether it improved production of trained scripts and promoted generalisation to untrained scripts, narrative speech and functional communication. METHODS: Two participants with aphasia received 20 hours of Script Therapy in EVA Park. Feasibility was assessed through session attendance, completion of practice and ratings of treatment fidelity. Acceptability was explored via post therapy interviews. The impact of therapy on script production, narrative production and functional communication was assessed through measures administered twice before therapy, immediately post therapy and at 5 weeks follow up. RESULTS: Participants attended at least 85% of sessions. Compliance with practice was good for one, but not the other. Fidelity ratings indicated that over 80% of core treatment components were fully present in recorded sessions. Participants expressed positive views about the intervention. Therapy significantly improved the production of words in trained scripts, with maintenance for one participant. Neither participant improved in the production of untrained scripts or personal narratives. One improved on the assessment of functional communication, but the margin of change was small. CONCLUSIONS: The study adds to the evidence that EVA Park can host a range of interventions and that this platform is acceptable to its intended user group.

Game Start in Forming Students’ Russian Speech Communication Skills

Games are an effective and entertaining way to develop students’ Russian speech communication skills. They not only improve your vocabulary, grammar and pronunciation, but also develop communication skills, the ability to express your thoughts and ideas, as well as interaction with others. According to the authors, the use of games in the educational process helps make learning more fun and interesting, and also contributes to a deeper understanding of the material. The authors came to the conclusion that the use of games in developing students’ Russian speech communication skills is an effective method that helps not only improve language skills, but also the development of communication skills and students’ enthusiasm for the educational process.

Vision Transformer Based Classification of Neurological Disorders from Human Speech

In this study, we introduce a transformative approach to achieve high-accuracy classification of distinct health categories, including Parkinson's disease, Multiple Sclerosis (MS), healthy individuals, and other categories, utilizing a transformer-based neural network. The cornerstone of this approach lies in the innovative conversion of human speech into spectrograms, which are subsequently transformed into visual images. This transformation process enables our network to capture intricate vocal patterns and subtle nuances that are indicative of various health conditions. The experimental validation of our approach underscores its remarkable performance, achieving exceptional accuracy in differentiating Parkinson's disease, MS, healthy subjects, and other categories. This breakthrough opens doors to potential clinical applications, offering an innovative, non-invasive diagnostic tool that rests on the fusion of spectrogram analysis and transformer-based models.

Method for testing the stability of an autoregressive model of the vocal tract and adjusting its parameters

Within the framework of the traditional direction of research in the field of acoustic measurements, an autoregressive model of the vocal tract as a key link in the human speech apparatus is considered. The acute problem of ensuring the stability of the autoregressive model in systems with adaptation of its parameters to the observed speech signal of short duration is pointed out. To overcome this problem, the task was set of testing the stability of the autoregressive model and adjusting its parameters based on the results of this testing. The study is based on the author’s method of formant analysis of vowel sounds of speech through the synthesis of a recursive shaping filter in the free oscillation mode. To solve sated task, a method is proposed for testing the stability and adjusting the parameters of the autoregressive model of the vocal tract based on a two-stage algorithm for its transformation. At the first stage of transformation, the stability of the autoregressive model is tested using the impulse response of the shaping filter. At the second stage, if the stability of the autoregressive model is violated, its impulse response is modified by element-by-element multiplication by a variable exponential value that asymptotically converges to zero. A regular algorithm has been developed for recalculating the modified impulse response into an adjusted vector of autoregressive parameters at the second stage of transformation. Based on the results of experimental testing of the proposed method, it was concluded that guaranteed stability of the autoregressive model of the vocal tract has been achieved with minimal distortion in the frequency domain. The results obtained are useful in the development and modernization of automatic speech recognition systems, digital speech communications, artificial intelligence and other information systems that use data compression and speech coding based on an autoregressive model of the vocal tract in automatic speech signal processing.

Активизация коммуникативной деятельности младших школьников на уроках окружающего мира (Часть 2)

The article proposes for discussion the contribution of primary schoolchildren studying the subject “The World Around Us” to the development of their speech and verbal communication. Based on the requirements of the Federal State Educational Standard for Primary General Education and the Federal Educational Program for Primary General Education, the methodology for developing dialogic speech (educational dialogues, didactic games), as well as descriptions and narratives in younger schoolchildren is revealed.

The expressiveness of human speech and voice: on the 95th anniversary of the birth of V.P. Morozov

The article analyzes the professional path of Vladimir Petrovich Morozov, the author of fundamental works in the field of psychophysiology and psychoacoustics of speech and voice, non-verbal communication, scientific foundations of vocal art. The main directions of his scientific activity are highlighted, their connection with the trends in the development of modern science, high relevance and interdisciplinary demand are shown. The special style of Morozov’s scientific creativity is emphasized, manifested in a combination of original theoretical ideas with the solution of practical problems, addressing borderline issues of psychology and musical art. The most important milestones of the scientist’s professional path and achievements in the methodology and theory of speech and voice psychology, as well as in experimental research, are described.

Креативность прецедентов: от слова к тексту. Границы актуализации (на материале английского и русского языков)

The aim of the study is to describe the ways of creating precedents in language, the possibilities of their realization, characteristic of ethnic language and culture. The article considers the situations of using precedent units in speech communication on the material of English and Russian languages. The analysis of the use of precedent units revealed the creative potential of the phenomenon of precedence and identified a number of opportunities for further study of the phenomenon in the activity aspect. The scientific novelty of the research consists in the fact that for the first time creative possibilities of precedent language units are considered within the framework of the system-activity approach to the study of speech communication in linguistic, cognitive and communicative dimensions. As a result of the study it was established that the creative process of making a precedent unit is an individual cognitive-communicative activity of each of the communicators. Highly communicative communicators are most active in the generation of creative texts; precedence in speech is justified by the framework of conjugation of cognitive activity, communicative activity of the author and cognitive activity, communicative activity of the recipient of speech communication.

Self-healing and reprocessable biobased non-isocyanate polyurethane elastomer with dual dynamic covalent adaptive network for flexible strain sensor

Owing to the nontoxicity of monomers, non-isocyanate polyurethane elastomers have attracted considerable attention, yet the synthesis and functionalization are still challenging. Herein, self-healing and reprocessable biobased non-isocyanate polyurethane (NIPU) elastomers with dual dynamic covalent adaptive network (DDCAN) by disulfide bonds and dynamic imine bonds were synthesized as substrate for the construction of flexible strain sensor with MXene as conductive substance. The tensile strength and elongation at break of the NIPU elastomer were 3.22 MPa and 234 %, respectively. Thanks to the formation of DDCAN, the NIPU elastomer showed high healing efficiency of 97.5 % after healing at room temperature for 24 h. Interestingly, the NIPU elastomer possessed superior reprocessing capability and the tensile strength after three reprocessing cycles reached 136.1 % of the original one, which was attributed to the increase of crosslinking points caused by topological rearrangement. In addition, the NIPU elastomer-based flexible strain sensor exhibited fast response (response time = 60 ms) and excellent repeatability (1000 bending-releasing cycles) and was successfully applied for human motion detection and speech recognition. The findings in this work conceivably stand out as a new methodology for the preparation of biobased and functional elastomers, which will greatly promote the sustainable development and application of flexible electronics.

An Enhanced Human Speech Based Emotion Recognition

An Enhanced Human Speech Based Emotion Recognition

Linguistic intuition in the system of teaching Russian as a foreign language at the pre-university stage

The relevance of the study is due to the lack of methodological complexes designed to develop linguistic intuition among foreigners studying the Russian language at pre-university stage. Foreign students at the initial level of education usually use linguistic intuition formed in their native language, but observations show that a full-fledged linguistic intuition in Russian-language speech communication is based on the automatic use of elements and constructions of the Russian language. The purpose of the study is to substantiate the need for consistent use of tasks that contribute to the automated speech skills and stimulate linguistic guess in the process of teaching Russian as foreign language. The following methods were applied: observation of the communicative activities of students learning the Russian language with the proposed methodology, analysis and synthesis of results, as well as observation in order to identify difficulties in achieving automaticity of speech skills and the use of linguistic intuition. The research was conducted at the elementary and basic levels of teaching the Russian language to Iranian students of medical and biological specialties at the Preparatory Faculty of the Kazan (Volga Region) Federal University. A methodology has been developed, based on the system of phonetic, lexical-grammatical and text tasks for classroom and independent work. The training complex uses elements of critical thinking to form and improve speech and cognitive skills that will help foreign students develop the ability to independently solve various communicative problems. Provided that tasks for developing intuition are regularly completed at classroom, it seems promising to achieve freedom of speech behavior of foreign students in communicative acts of various types.

Voice interactive information metasurface system for simultaneous wireless information transmission and power transfer

Intelligent voice interaction offers a flexible and powerful way to connect individuals with smart devices beyond our expectations. The real-time nature of voice communication enables smart devices to comprehend the user language, execute the corresponding instructions, and facilitate seamless communications, transforming our lives in unprecedented ways. Owing to self-adaptive and reprogrammable functionalities, information metasurface (IMS) opens up a new avenue for smart home and smart cities. To further enhance the intelligence of IMS, we propose an IMS system via intelligent voice interaction and information processing. The voice interaction enables the efficient remote control on the IMS in a flexible, convenient, touchless manner. Leveraging speech recognition, speech synthesis, target detection, and communication technologies, the IMS system achieves automatic beam manipulation capabilities for wireless information transmissions and wireless power transfers. The IMS system is designed to operate in two distinct modes: instruction mode, wherein the user instructs the operations, and autonomous mode, wherein the automatic detections govern the actions, in which seamless mode switching through the voice commands is supported. Users can flexibly achieve precise control over the functions of the intelligent metasurface system through voice interaction at a distance, without the need for close-range manual touch control, which greatly simplifies the operation difficulty and is particularly suitable for remote control and complex application scenarios. A series of experiments, including wireless video transmissions and wireless power transfers are conducted to demonstrate the flexibility and convenience of the IMS system. The incorporation of intelligent voice interaction technology with the IMS presents a novel paradigm for the applications of programmable and information metasurfaces.

Emotions recognition in audio signals using an extension of the latent block model

Emotion detection in human speech is a significant area of research, crucial for various applications such as affective computing and human–computer interaction. Despite advancements, accurately categorizing emotional states in speech remains challenging due to its subjective nature and the complexity of human emotions. To address this, we propose leveraging Mel frequency cepstral coefficients (MFCCS) and extend the latent block model (LBM) probabilistic clustering technique with a Gaussian multi-way latent block model (GMWLBM). Our objective is to categorize speech emotions into coherent groups based on the emotional states conveyed by speakers. We employ MFCCS from time-series audio data and utilize a variational Expectation Maximization method to estimate GMWLBM parameters. Additionally, we introduce an integrated Classification Likelihood (ICL) model selection criterion to determine the optimal number of clusters, enhancing robustness. Numerical experiments on real data from the Berlin Database of Emotional Speech (EMO-DB) demonstrate our method’s efficacy in accurately detecting and classifying emotional states in human speech, even in challenging real-world scenarios, thereby contributing significantly to affective computing and human–computer interaction applications.