Speech System Research Articles

Machine learning for predictive AAC: Improving speech and gesture-based communication systems

Augmentative and alternative communication (AAC) systems play a crucial role in supporting individuals with severe communication disabilities by providing accessible means of expression and engagement. However, many conventional AAC devices rely on manual input or basic predictive functions, which can limit communication efficiency and responsiveness. The application of machine learning (ML) to AAC offers new opportunities to enhance these systems, enabling them to provide faster, more accurate, and contextually relevant communication assistance. Advances in ML, particularly in predictive text, speech recognition, and gesture interpretation, allow AAC systems to adapt more intuitively to user needs, predicting intent based on usage patterns and multimodal data, such as voice and gestures. Current research highlights the potential of ML to address key gaps in AAC technology by creating more responsive, personalized systems that align with individual user behaviours. This study proposes a novel ML framework designed to integrate these capabilities, promising improvements in communication speed, user autonomy, and accuracy. By addressing the challenges and limitations of traditional AAC devices, this research aims to advance accessible communication solutions that empower users and improve quality of life.

THE IMPACT OF ARTIFICIAL INTELLIGENCE ON ENGLISH LANGUAGE TEACHING IN THE CONTEXT OF FORMAL AND INFORMAL HIGHER EDUCATION

This article considersthe impactof artificial intelligence on the process of teaching English in formal and informal higher education. The authors studythe use of various artificial intelligence technologies, such as machine learning, neural networks and automated speech analysis systems, to improve the effectiveness of teaching English to students, analyze the advantages and disadvantages of using artificial intelligence in teaching English, and offer recommendations for optimizing this process. The authors presentedresearchof the influence of artificial intelligence on the process of teaching English through a theoretical review of foreign experience and a questionnaire survey, the purpose of which was to study the opinions and views of students on the influence of artificial intelligence on the process of teaching English. The results of the researchmay be useful for teachers and students planning to introduceartificial intelligence into English language teaching.

FPGA-Based Design of a Ready-to-Use and Configurable Soft IP Core for Frame Blocking Time-Sampled Digital Speech Signals

‘Frame blocking’ or ‘Framing’ is a technique that divides a time-sampled speech or audio signal into consecutive and equi-sized short-time frames, either overlapped or non-overlapped, for analysis. The framing hardware architectures (FHA) in the literature support framing speech or audio samples of specific word size with specific frame size and frame overlap size. However, speech and audio applications often require framing signal samples of varied word sizes with varied frame sizes and frame overlap sizes. Therefore, the existing FHAs must be redesigned appropriately to keep up with the variability in word size, frame size and frame overlap size, as demanded across multiple applications. Redesigning the existing FHAs for each specific application is laborious, prompting the need for a configurable intellectual property (IP) core. The existing FHAs are inappropriate for creating configurable IP cores as they lack adaptability to accommodate variability in frame size and frame overlap size. Therefore, to address these issues, a novel FHA, adaptable to accommodate the desired variability, is proposed. Furthermore, the proposed FHA is transformed into a field-programmable gate array-based soft, ready-to-use and configurable frame blocking IP core using the Xilinx® Vivado™ tool. The resulting IP core is versatile, offering configurability for framing in numerous applications incorporating real-time digital speech and audio systems. This research article discusses the proposed FHA and frame blocking IP core in detail.

“Idol talks!” AI-driven image to text to speech: illustrated by an application to images of deities

This work aims to provide an innovative solution to enhance the accessibility of images by an innovative image to text to speech system. It is applied to Hindu and Christian divine images. The method is applicable, among others, to enhance cultural understanding of these images by the visually impaired. The proposed system utilizes advanced object detection techniques like YOLO V5 and caption generation techniques like ensemble models. The system accurately identifies significant objects in images of Deities. These objects are then translated into descriptive and culturally relevant text through a Google text-to-speech synthesis module. The incorporation of text generation techniques from images introduces a new perspective to the proposed work. The aim is to provide a more comprehensive understanding of the visual content and allow visually impaired individuals to connect with the spiritual elements of deities through the immersive experience of auditory perception through a multimodal approach to make them feel inclusive in the community. This work is also applicable to preserve Cultural Heritage, Tourism and integrating with Virtual Reality (VR) and Augmented Reality (AR). Images of the artistic cultural legacy are hardly available in annotated databases, particularly those featuring idols. So we gathered, transcribed, and created a new database of Religious Idols in order to satisfy this requirement. In this paper, we experimented how to handle an issue of religious idol recognition using deep neural networks. In order to achieve this outcome, the network is first pre-trained on various deep learning models, and the best one which outperforms others is chosen. The proposed model achieves an accuracy of 96.75% for idol detection, and an approximate 97.06% accuracy for text generation according to the BLEU score.

On a new approach to studying lexical metonymy. Review of a monograph: Ilyukhina N.A. Lexical metonymy in linguistic and cognitive comprehension. Samara: OOO «SAMARAMA», 2023, 172 p. ISBN 978-5-6050869-9-4

This review is devoted to the monograph «Lexical metonymy in linguistic and cognitive comprehension» by N.A. Ilyukhina, who examines the problems of using metonymic nomination of objects, processes and persons in various denotative and discursive spheres. The relevance of the issues raised in the monograph is determined by the important role of associative thinking in the process of understanding the reality and complexity of mental connections between referents, which become the objects of metonymic nomination. The review describes the issues that have become the focus of the researcher’s attention, in particular, the typologization of lexical metonymy, substantive and verbal metonymy, metonymic transfer of definitions, metonymy in various denotative and discursive spheres. The problem of the research lies in the complexity of the linguistic representation of the cognitive mechanisms of metonymic transfer, in understanding the mental connections between the part and the whole, due to which their names can replace each other in language and speech. During the research N.A. Ilyukhina used an up-to-date method of linguocognitive analysis. As a result of the conducted fundamental research, N.A. Ilyukhina comes to a number of important conclusions. The most significant and interesting conclusions are about the unified cognitive foundations of the main transfers in the system of language and speech; about the connection of the transfer vector with the structural type of a multi-faceted concept, which contains knowledge about the corresponding referent, nominated or defined by the word; about the most significant role of the cognitive metonymic mechanism in speech activity; about the «ubiquity», universality, versatility and expansion of linguistic metonymy due to its cognitive foundations; about the possibility of using metonymy as a basis for other tropes. The monograph is of considerable interest to researchers dealing with the problems of metonymy, cognitive linguistics, semasiology, and semantics of discourse.

Bayesian inference of state feedback control parameters for fo perturbation responses in cerebellar ataxia.

Behavioral speech tasks have been widely used to understand the mechanisms of speech motor control in typical speakers as well as in various clinical populations. However, determining which neural functions differ between typical speakers and clinical populations based on behavioral data alone is difficult because multiple mechanisms may lead to the same behavioral differences. For example, individuals with cerebellar ataxia (CA) produce atypically large compensatory responses to pitch perturbations in their auditory feedback, compared to typical speakers, but this pattern could have many explanations. Here, computational modeling techniques were used to address this challenge. Bayesian inference was used to fit a state feedback control (SFC) model of voice fundamental frequency (fo) control to the behavioral pitch perturbation responses of speakers with CA and typical speakers. This fitting process resulted in estimates of posterior likelihood distributions for five model parameters (sensory feedback delays, absolute and relative levels of auditory and somatosensory feedback noise, and controller gain), which were compared between the two groups. Results suggest that the speakers with CA may proportionally weight auditory and somatosensory feedback differently from typical speakers. Specifically, the CA group showed a greater relative sensitivity to auditory feedback than the control group. There were also large group differences in the controller gain parameter, suggesting increased motor output responses to target errors in the CA group. These modeling results generate hypotheses about how CA may affect the speech motor system, which could help guide future empirical investigations in CA. This study also demonstrates the overall proof-of-principle of using this Bayesian inference approach to understand behavioral speech data in terms of interpretable parameters of speech motor control models.

Developmental Aspects of Greek Vowel Reduction in Different Prosodic Positions

This study investigates the development of Greek vowel reduction across different prosodic positions (stressed, pre-stressed, post-stressed), examining normative data from 72 participants aged 3 years to adulthood and balanced for gender. Participants performed a delayed repetition task, producing real trisyllabic words with the vowels [i, ε, ɐ, o, u] examined in the second syllable. Measurements included relative vowel duration, normalized acoustic vowel space areas, and Euclidean distances of vowels from the centroid of the acoustic space. Our findings show that changes in speech motor control, system stiffness, and stress marking with age, along with children’s prosody sensitivity, contributed to several developmental milestones: the completion of the developmental trajectory of relative vowel duration and temporal vowel reduction at early adolescence; the attainment of adult-like spatial vowel characteristics and their reduction at preschool age; and the early acquisition of the prosodic strength of the stress conditions, leading to vowel reduction from the stressed to pre-stressed to post-stressed conditions. The correlation strength between temporal and spatial vowel reduction across ages revealed age-related differences in spatiotemporal speech organization, with significant gender-related differences observed only in vowel space areas, where females exhibited larger areas possibly related to sociophonetic factors. Intrinsic vowel duration appeared from age 3.

A low-complexity adaptive speech dereverberation approach based on multichannel linear prediction

The adaptive multichannel linear prediction (AMCLP) technique is one of the most effective methods for speech dereverberation in real-time application systems. However, the computational complexity of the associated AMCLP algorithms is very high. In this paper, we propose a low-complexity speech dereverberation approach based on AMCLP. The coefficient matrix of the multichannel dereverberation filter is decomposed into a linear combination of a set of low-dimensional sub-filter matrices and a set of low-dimensional sub-filter vectors through Kronecker product. According to these sub-filter models, we define two sets of signal models and cost functions, from which the adaptive speech dereverberation algorithm is established. This solution not only leads to effective dereverberation performance, but also reduces the computational cost, which is very conducive to realistic speech systems.

Sensorimotor adaptation to a non-uniform formant perturbation generalizes to untrained vowels.

When speakers learn to change the way they produce a speech sound, how much does that learning generalize to other speech sounds? Past studies of speech sensorimotor learning have typically tested the generalization of a single transformation learned in a single context. Here, we investigate the ability of the speech motor system to generalize learning when multiple opposing sensorimotor transformations are learned in separate regions of the vowel space. We find that speakers adapt to a non-uniform "centralization" perturbation, learning to produce vowels with greater acoustic contrast, and that this adaptation generalizes to untrained vowels, which pattern like neighboring trained vowels and show increased contrast of a similar magnitude.

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

The effect of different visual feedback interfaces of music training games on speech rehabilitation in hearing-impaired children: An fNIRS study

Singing plays a critical role in enhancing musicality, sound discrimination, and attention, and proves advantageous for speech rehabilitation in children with hearing impairments. Computer-based training games are well-suited to the learning behaviors of children, with substantial evidence suggesting that music training augments speech training capabilities in this demographic. Despite this, there is a lack of detailed exploration into the design of interactive online music training interfaces tailored for these needs. This study investigates brain activation changes using two visual feedback singing games, analyzed through functional near-infrared spectroscopy: a serious game (SG) and an entertainment game (EG) with visually enhanced feedback. It also assesses the efficacy of home-based music training software for speech rehabilitation.Methods involved recording oxygenated hemoglobin concentration (Delta [HbO]) signals from the prefrontal cortex, motor cortex, occipital lobe, and temporal lobe in 21 children (average age: 9.3 ± 1.9 years) during two singing interface experiments. Subjects also completed the Intrinsic Motivation Inventory (IMI) questionnaire post-experiment.Main results showed that brain regions, particularly the temporal lobe, exhibited stronger and more pronounced activation signals with the SG interface compared to the EG, suggesting that SG is more effective for speech system rehabilitation. The Intrinsic Motivation Scale results revealed higher acceptability for SG than for EG. This study provides insights into designing online speech rehabilitation products for children with hearing impairment, advocating for better interactive training methods from a neuroscience perspective.

Engagement of the speech motor system in challenging speech perception: Activation likelihood estimation meta-analyses.

The relationship between speech production and perception is a topic of ongoing debate. Some argue that there is little interaction between the two, while others claim they share representations and processes. One perspective suggests increased recruitment of the speech motor system in demanding listening situations to facilitate perception. However, uncertainties persist regarding the specific regions involved and the listening conditions influencing its engagement. This study used activation likelihood estimation in coordinate-based meta-analyses to investigate the neural overlap between speech production and three speech perception conditions: speech-in-noise, spectrally degraded speech and linguistically complex speech. Neural overlap was observed in the left frontal, insular and temporal regions. Key nodes included the left frontal operculum (FOC), left posterior lateral part of the inferior frontal gyrus (IFG), left planum temporale (PT), and left pre-supplementary motor area (pre-SMA). The left IFG activation was consistently observed during linguistic processing, suggesting sensitivity to the linguistic content of speech. In comparison, the left pre-SMA activation was observed when processing degraded and noisy signals, indicating sensitivity to signal quality. Activations of the left PT and FOC activation were noted in all conditions, with the posterior FOC area overlapping in all conditions. Our meta-analysis reveals context-independent (FOC, PT) and context-dependent (pre-SMA, posterior lateral IFG) regions within the speech motor system during challenging speech perception. These regions could contribute to sensorimotor integration and executive cognitive control for perception and production.

Identifying data elements and key features to design a telerehabilitation system for speech and language disorders in children with hearing impairments in Iran: A cross-sectional study.

Designing and implementing telerehabilitation systems would require the identification of data elements to meet disabled people's needs. An in-depth study on the identification and validation of data elements can help design and implement telerehabilitation systems successfully. Therefore, this study aimed to identify and validate data elements to design a telerehabilitation system for the speech and language disorders of hearing-impaired children in Iran. This descriptive cross-sectional study was conducted in three steps including literature review, focused group discussion, and Delphi technique implementation to extract and validate data elements. In the first step, the literature on electronic databases was reviewed to extract the data elements of telerehabilitation systems, and nine studies were selected based on the inclusion criteria. In the second step, a focused group discussion was held to review and classify the extracted data elements. Finally, the Delphi technique was employed to validate the drafted data elements. In total, 352 data elements were extracted from the literature review. Finally, 102 data elements in 10 categories (Demographic and Clinical information of the disabled person, Clinical history, Demographic information of the provider, Customization of exercises, Reminders, Online and offline counseling, and training, Reporting, Key features of the system, Evaluation of the progress of the disabled person) were classified and validated by experts as essential data elements to design a telerehabilitation system for the speech and language disorders of hearing-impaired children. The necessary data elements were proposed as the foundations to design a telerehabilitation system for the speech and language disorders of hearing-impaired children. These data elements help design and implement telerehabilitation systems successfully so that such systems can easily be provided for children with hearing disabilities.

Hierarchical Sentiment Analysis Framework for Hate Speech Detection: Implementing Binary and Multiclass Classification Strategy

A significant challenge in automating hate speech detection on social media is distinguishing hate speech from regular and offensive language. These identify an essential category of content that web filters seek to remove. Only automated methods can manage this volume of daily data. To solve this problem, the community of Natural Language Processing is currently investigating different ways of hate speech detection. In addition to those, previous approaches (e.g., Convolutional Neural Networks, multi-channel BERT models, and lexical detection) have always achieved low precision without carefully treating other related tasks like sentiment analysis and emotion classification. They still like to group all messages with specific words in them as hate speech simply because those terms often appear alongside hateful rhetoric. In this research, our paper presented the hate speech text classification system model drawn upon deep learning and machine learning. In this paper, we propose a new multitask model integrated with shared emotional representations to detect hate speech across the English language. The Transformer-based model we used from Hugging Face and sentiment analysis helped us prevent false positives. Conclusion. We conclude that utilizing sentiment analysis and a Transformer-based trained model considerably improves hate speech detection across multiple datasets.

MKELM based multi-classification model for foreign accent identification

The automatic identification of foreign accents can play a crucial role in various speech systems, including speaker identification, e-learning, telephone banking, and more. Additionally, it can greatly enhance the robustness of Automatic Speech Recognition (ASR) systems. Non-native accents in speech signals are characterized by distinct pronunciations, prosody, and voice characteristics of the speaker. However, automatically identifying foreign accents poses significant challenges, particularly in the context of multi-class modeling. Multi-classification models face difficulties in achieving high performance and dealing with computational challenges when confronted with multi-dimensional and unbalanced datasets, such as those with more than two accents. Furthermore, the choice of features remains a bottleneck problem for Foreign Accent Identification (FAID), further hindering performance in these tasks. Consequently, the accuracy of current systems is typically low. To address these challenges, this paper proposes a framework based on the Multi-Kernel Extreme Learning Machine (MKELM) model for the multi-classification of FAID. The MKELM model utilizes a novel weighted scheme to classify various non-native English accents, including Arabic, Chinese, Korean, French, and Spanish. The model first combines Mel-frequency cepstral coefficients (MFCCs) and prosodic features as input, trains pairwise binary classifiers independently, and subsequently employs a weighting scheme to distinguish between classes and identify accents. Through experiments, the proposed model achieves an accuracy rate of 84.72% using a paired weighting scheme. In contrast, the accuracy rate drops to 66.5% when employing the traditional non-weighted multi-classification scheme. A comparison with other models demonstrates the significant advantages of the proposed model in FAID multi-class classification, showcasing improved accuracy, reduced computational complexity (requiring fewer computations, faster learning rates, and shorter training time), and enhanced stability compared to state-of-the-art classification methods.

Exploring the Impact of Fine-Tuning the Wav2vec2 Model in Database-Independent Detection of Dysarthric Speech.

Many acoustic features and machine learning models have been studied to build automatic detection systems to distinguish dysarthric speech from healthy speech. These systems can help to improve the reliability of diagnosis. However, speech recorded for diagnosis in real-life clinical conditions can differ from the training data of the detection system in terms of, for example, recording conditions, speaker identity, and language. These mismatches may lead to a reduction in detection performance in practical applications. In this study, we investigate the use of the wav2vec2 model as a feature extractor together with a support vector machine (SVM) classifier to build automatic detection systems for dysarthric speech. The performance of the wav2vec2 features is evaluated in two cross-database scenarios, language-dependent and language-independent, to study their generalizability to unseen speakers, recording conditions, and languages before and after fine-tuning the wav2vec2 model. The results revealed that the fine-tuned wav2vec2 features showed better generalization in both scenarios and gave an absolute accuracy improvement of 1.46%-8.65% compared to the non-fine-tuned wav2vec2 features.

Speaker-independent speech inversion for recovery of velopharyngeal port constriction degreea).

For most of his illustrious career, Ken Stevens focused on examining and documenting the rich detail about vocal tract changes available to listeners underlying the acoustic signal of speech. Current approaches to speech inversion take advantage of this rich detail to recover information about articulatory movement. Our previous speech inversion work focused on movements of the tongue and lips, for which "ground truth" is readily available. In this study, we describe acquisition and validation of ground-truth articulatory data about velopharyngeal port constriction, using both the well-established measure of nasometry plus a novel technique-high-speed nasopharyngoscopy. Nasometry measures the acoustic output of the nasal and oral cavities to derive the measure nasalance. High-speed nasopharyngoscopy captures images of the nasopharyngeal region and can resolve velar motion during speech. By comparing simultaneously collected data from both acquisition modalities, we show that nasalance is a sufficiently sensitive measure to use as ground truth for our speech inversion system. Further, a speech inversion system trained on nasalance can recover known patterns of velopharyngeal port constriction shown by American English speakers. Our findings match well with Stevens' own studies of the acoustics of nasal consonants.

Multifunctional, High-Strength Electronic Skin Based on the Natural Sheepskin Fiber Network for Multifaceted Human Health Monitoring and Management.

Inspired by the animal skin fiber network, we developed an electronic skin (e-skin) utilizing natural sheepskin as the primary substrate. This innovative design addresses the limitations of conventional e-skins, including inadequate mechanical strength, overly complex artificial network construction, and limited health monitoring capabilities. This e-skin successfully retains the structure and properties of natural sheepskin while exhibiting exceptional mechanical strength (with a breaking strength of 4.01 MPa) and high elongation (with an elongation at a break of 304.8%). Moreover, it possesses various desirable attributes such as electrical conductivity, antibacterial properties, biocompatibility, and environmental stability. In addition, this e-skin has the advantage of diverse data collection (joint movement, bioelectricity, foot health detection, and speech disorder communication systems). Therefore, this e-skin breaks the traditional construction strategy and single-mode application and is expected to become an ideal material for building smart sensor devices.

Stability of ECoG high gamma signals during speech and implications for a speech BCI system in an individual with ALS: a year-long longitudinal study

Objective. Speech brain–computer interfaces (BCIs) have the potential to augment communication in individuals with impaired speech due to muscle weakness, for example in amyotrophic lateral sclerosis (ALS) and other neurological disorders. However, to achieve long-term, reliable use of a speech BCI, it is essential for speech-related neural signal changes to be stable over long periods of time. Here we study, for the first time, the stability of speech-related electrocorticographic (ECoG) signals recorded from a chronically implanted ECoG BCI over a 12 month period. Approach. ECoG signals were recorded by an ECoG array implanted over the ventral sensorimotor cortex in a clinical trial participant with ALS. Because ECoG-based speech decoding has most often relied on broadband high gamma (HG) signal changes relative to baseline (non-speech) conditions, we studied longitudinal changes of HG band power at baseline and during speech, and we compared these with residual high frequency noise levels at baseline. Stability was further assessed by longitudinal measurements of signal-to-noise ratio, activation ratio, and peak speech-related HG response magnitude (HG response peaks). Lastly, we analyzed the stability of the event-related HG power changes (HG responses) for individual syllables at each electrode. Main Results. We found that speech-related ECoG signal responses were stable over a range of syllables activating different articulators for the first year after implantation. Significance. Together, our results indicate that ECoG can be a stable recording modality for long-term speech BCI systems for those living with severe paralysis. Clinical Trial Information. ClinicalTrials.gov, registration number NCT03567213.

SEMANTIC FIELD “POWER” IN’S NOVEL “THE ARC FOR THE UNINVITED” BY VLADIMIR MAKSIMOV

The article presents a comprehensive description of the semantic field ‟Power” in the novel by the well-known Russian prose writer of the 2nd half of the 20th century, Vladimir Yemelyanovich Maksimov. The purpose of the study is to identify the structural and semantic features of the field, the communicative and pragmatic properties of its nuclear components, which are semantic dominants of the text and actively participate in organising the artistic narrative, depicting characters and events, and explicating the conceptual content of the study. An analytical review of the main linguistic studies devoted to the study of the phenomenon of authority in Russian culture and linguistic consciousness, and the scientific description of semantic fields in the system of language and speech is carried out. Modelling the field structure of the novel is based on the principles of discourse analysis of the text, and it is determined by the specifics of the writer’s linguistic personality, the individual originality of the artistic picture of the world, the genre-stylistic and plot-compositional features of the work, and the linguacultural situation of the depicted era. The component composition of the field, its core, centre and periphery, and microfields are consistently characterised. Semantic connections and intersections of field components are determined with contextual conditionality of additional meanings revealed. The semantic field ‟authority” combines four microfields: the central part of the field structure includes the microfields ‟state authority” and ‟God power”, the peripheral zone includes the microfields ‟family authority” and ‟authority and power of natural elements”. The core of the field includes the key lexemes of authority, power and the leader. The semantic dominants of the text, being components of the analysed field structure, contribute to the development of action in space and time, create narrative polyphony, reflect the author’s position, and convey the bright features of the writer’s linguistic personality.