Speech Content Research Articles

A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods

Affective computing is one of the most important research fields in modern human–computer interaction (HCI). The goal of affective computing is to study and develop the theories, methods, and systems that can recognize, explain, process, and simulate human emotions. As a branch of affective computing, emotion recognition aims to enlighten the machine/computer automatically analyzing human emotions, which has received increasing attention from researchers in various fields. Human beings generally observe and understand the emotional states of one person by integrating the perceived information from his/her facial expressions, voice tone, speech content, behavior, or physiological features. To imitate the emotion observation manner of humans, researchers have been devoted to constructing multimodal emotion recognition models by fusing information from two or more modalities. In this paper, we provide a comprehensive review of multimodal emotion recognition from the perspectives of multimodal datasets, data preprocessing, unimodal feature extraction, and multimodal information fusion methods in recent decades. Furthermore, challenges and future research directions of the topic are specified and discussed. The main motivations of this review are to conclude the recent emergence of abundant works on multimodal emotion recognition and to provide potential guidance to researchers in the related field for understanding the pipeline and mainstream approaches to multimodal emotion recognition.

Fanon's revolutionary murmurs: Toward a critical phenomenology of listening

AbstractDecolonial, postcolonial, and postmodern criticism is indebted to Frantz Fanon for revealing the hegemony of vision and visual language in Western imperial discourse. Yet, the import of Fanon's critique of coloniality reaches beyond a focus on vision into the sonorous. Attending to the often overlooked auditory dimension of Fanon's work, I argue, brings attention to the role of listening as a condition of possibility of a revolutionary consciousness. Listening to Fanon's careful descriptions of the experience of many Algerians of listening to the National Liberation Front's radio broadcast, The Voice of Free and Fighting Algeria in “This is the Voice of Algeria,” I propose that what makes possible the institution of a revolutionary consciousness is listening to murmurs. Listening to the materiality of voice and sound rather than the propositional content of speech is revolutionary because it challenges core values central to the project of colonial modernity: metaphysical logocentrism and the fungibility to which colonial modernity strives to reduce colonized subjects.

Silent speech recognition using data augmentation based on a three-dimensional lip model

In this study, we proposed a data augmentation method using a three-dimensional model of a speaker's face for machine lip reading, which estimates the content of speech without using speech data. The proposed method converted a speaker's face into a three-dimensional model using DECA (Detailed Expression Capture and Animation), and generated a large amount of learning data by rotating the three-dimensional model in different directions. Then we obtained the facial features around the lips as time series data using dlib, which is an OpenCV module. The time series data is fed into a recognition model. We introduced end-to-end recognition model which is used in continuous speech recognition. The recognition model is based on DeepSpeech2, and we modified the filter size of CNN (convolutional neural network) and the number of layers of BiGRU (bidirectional gated recurrent unit) to adapt to our goal. We evaluated the proposed method on ten ordinary Japanese words. We used the phoneme error rate as the evaluation index. As a result of the evaluation, the error rate of about 0.32 was achieved even for data where the speaker of the evaluation data was not included in the speaker of the training data.

CODE MIXING USED IN TASYA FARASYA BEAUTY VLOGS

This study entitled “Code Mixing Used In Tasya Farasya Beauty Vlogs”. Beauty vloggers often share information about the cosmetic products they use or review beauty and make-up products. Sometimes they mix several languages in their videos, such as mixing English with Indonesian. Language has a very important role in our life. We use language for communicating and people nowadays speak in multiple languages. According to the phenomenon above, this research focuses on code mixing in one of Tasya Farasya's beauty vlogs on her YouTube channel, level of code mixing and to find the reasons why Tasya mixed her language. There are two theories applied in this study, namely the theory of the types of code mixing and reason of code mixing proposed by Hoffman (1991) and the theory of the level of code mixing proposed by Suwito (1996). This research is the qualitative research. The methods applied in collecting the data are the documentation and observing method and by using the note-taking technique. The data is taken mainly from the utterances found in the Tasya Farasya beauty vlogs, as the main data source for this study. The result of this study showed that there are three types of code mixing found used by Tasya videos. There are data collected for the type of code mixing. For the level of code mixing, and for the reason of code mixing. Type of code mixing there are intra-sentential, intra-lexical, involvement in a change of pronunciation. And for level of code mixing there are word level, phrase level, baster level, repetition level, idiom level, clause level. And for the reason of code mixing there are talking about particular topic, being emphatic about something, interjection, repetition used for clarification, expressing group identity, clarifying the speech content for the interlocutor.

Political Hate Speech Detection using Machine Learning

In the digital age, the detection of hate speech has become a crucial task, given the pervasive and detrimental impact of offensive content on individuals and communities. This project is dedicated to developing a machine learning-driven approach for the identification and mitigation of hate speech within online platforms.The primary aim of this research is to explore the effectiveness of various machine learning algorithms in the accurate and efficient detection of hate speech in political speeches. The project entails the collection and preprocessing of a substantial dataset consisting of user- generated content from social media platforms and online forums, with annotations indicating hate speech content. The dataset is then partitioned into training, validation, and testing sets, and natural language processing techniques are employed to extract relevant features. Diverse machine learning models, including logistic regression, support vector machines are implemented and fine-tuned using the training data. In summary, this contributes to the ongoing fight against hate speech in political speeches by harnessing the power of machine learning. The findings not only shed light on the challenges and potential solutions in hate speech detection but also pave the way for the development of more robust and precise systems, ultimately fostering a safer and more inclusive online environment. Key Words: Hate Speech Detection, Political Hate Speech , Machine Learning, Offensive Content , Support Vector Machine.

FORMATION OF LANGUAGE:PSYCHOLOGICAL AND PEDAGOGICAL ASPECTS

In child development, language plays a crucial role in the processes of differentiation and reintegration into new situational forms. These forms include "others," "language for others," "themselves," and "language for themselves," among others. Language is both a process and a product that gains unique sense and character through joint activity between parents and the child. The structure, functions, and contents of indirect speech act as the basis for individual consciousness. Some researchers have provided evidence that the dialectics of communication with others and oneself is central to the conscious self-control of children.According to L. Vygotsky, the language initially acquired by a child is not an independent version of grammar but a means of communication created through interactions between parents and the child. A. Luria's research indicates that for all higher cognitive processes, it is necessary to transition from external social experience to internal psychological control. In other words, the development of cognitive control requires the shift from relying on social experiences to relying on internal psychological processes.

Demographical Based Sentiment Analysis for Detection of Hate Speech Tweets for Low Resource Language

Advancement in IT and communication technology provides the opportunity for social media users to communicate their ideas and thoughts across the globe within no time as well big data promulgated in a result of the communication process itself has immense challenges. Recently, the provision of freedom of speech has witnessed immense promulgation of offensive and hate speech content on the internet aimed the basic human rights violation. The detection of abusive content on social media for rich resource language has become a hot area for researchers in the recent past. However, low-resource languages are underprivileged due to the non-availability of large corpus and its complexity to understand. The proposed methodology mainly has two parts. One is to detect abusive content and the other is to have a demographical analysis of the Indigenously developed dataset. The process starts with the development of a unique unlabeled Urdu dataset of 0.2 M from Twitter through a web scrapper tool named snscraper. The dataset is collected against the 36 districts of Punjab from Pakistan and from the duration 2018- Apr 2022. The dataset is labeled into three target classes Neutral, Offensive, and Hate Speech. After data cleaning, the feature extraction process is achieved with the help of traditional techniques such as Bow and tf-idf with the combination of word and char n-gram and word embedding word2Vec. The dataset is trained on both machine learning algorithms SVM and Logistic regression and deep learning techniques Long Short Term Memory (LSTM) and Convolutional Neural Networks (CNN). The best F score achieved through LSTM on this dataset is 64 and accuracy is 93 through CNN algorithms. A Choropleth map is used for visualization of the dataset distributed among 36 districts of Punjab and a time series plot for time analysis covers five years duration from 2018-Apr to 22.

A comparison of linguistic patterns between individuals with current major depressive disorder, past major depressive disorder, and controls in a virtual, psychiatric research interview

Major Depressive Disorder (MDD) is a leading health burden worldwide. Previous research has demonstrated that linguistic analysis of depressed individuals’ written and oral speech has potential as a diagnostic and monitoring biomarker. We sought to determine if the semantic content of speech differs between individuals with current MDD, past MDD, and controls. We recruited 53 volunteers for a simulated telehealth psychiatric intake interview. The sample included 14 individuals with current MDD, 21 with past MDD, and 18controls, all confirmed using a semi-structured diagnostic interview. The manually-transcribed interview transcripts were analyzed utilizing the LIWC-22 dictionary and statistical tests were applied to identify differences in the linguistic patterns between each clinical categorization. When comparing depressed subjects (either current or past) versus controls, significant differences were found for emotional tone, total function words, auxiliary verbs, negative tone, negative emotion, anxiety, sadness, attention, and visual. Individuals with past MDD only differed from those with current MDD in use of analytical thinking and auxiliary verbs. These results indicate that LIWC categories could differentiate current or past depressed subjects from controls, but fewer differences emerged when comparing current and past MDD. Further prospective studies with larger sample sizes are needed to confirm these findings.

AN ANALYSIS OF INDONESIAN-ENGLISH CODE-MIXING FOUND IN BANTEN

This research intends to analysis the types of code-mixing in Banten employed by 2nd grade students of SMAN 2 Kota Serang, to know the factors of students used code-mixing, to know the reasons of students used code-mixing. The mixed method was used. To obtain the data observation, questionnaire, and interview were employed. The result showed all 2nd grade students of XI MIPA 4 used code mixing. The types of code mixing used by students were morphemes, word, modifiers, phrase, clause, sentences. The factors of students doing code mixing were participation roles and relationship, situational factor, message intrinsic factor, and language attitude, dominance attitude, and security attitude factors. While the reasons that cause the students used code mixing were talking about certain topics, quoting others, be firm about something (express solidarity), interjection (inserting sentence filler or sentence conjunction), repetition is used for communication, intention to clarify speech content or speech, expressing group identity. Therefore, It is visible the students used code mixing in their activities.

The Role of Word Content, Sentence Information, and Vocoding for Voice Cue Perception.

For voice perception, two voice cues, the fundamental frequency (fo) and/or vocal tract length (VTL), seem to largely contribute to identification of voices and speaker characteristics. Acoustic content related to these voice cues is altered in cochlear implant transmitted speech, rendering voice perception difficult for the implant user. In everyday listening, there could be some facilitation from top-down compensatory mechanisms such as from use of linguistic content. Recently, we have shown a lexical content benefit on just-noticeable differences (JNDs) in VTL perception, which was not affected by vocoding. Whether this observed benefit relates to lexicality or phonemic content and whether additional sentence information can affect voice cue perception as well were investigated in this study. This study examined lexical benefit on VTL perception, by comparing words, time-reversed words, and nonwords, to investigate the contribution of lexical (words vs. nonwords) or phonetic (nonwords vs. reversed words) information. In addition, we investigated the effect of amount of speech (auditory) information on fo and VTL voice cue perception, by comparing words to sentences. In both experiments, nonvocoded and vocoded auditory stimuli were presented. The outcomes showed a replication of the detrimental effect reversed words have on VTL perception. Smaller JNDs were shown for stimuli containing lexical and/or phonemic information. Experiment 2 showed a benefit in processing full sentences compared to single words in both fo and VTL perception. In both experiments, there was an effect of vocoding, which only interacted with sentence information for fo. In addition to previous findings suggesting a lexical benefit, the current results show, more specifically, that lexical and phonemic information improves VTL perception. fo and VTL perception benefits from more sentence information compared to words. These results indicate that cochlear implant users may be able to partially compensate for voice cue perception difficulties by relying on the linguistic content and rich acoustic cues of everyday speech. https://doi.org/10.23641/asha.23796405.

GestureLens: Visual Analysis of Gestures in Presentation Videos.

Appropriate gestures can enhance message delivery and audience engagement in both daily communication and public presentations. In this paper, we contribute a visual analytic approach that assists professional public speaking coaches in improving their practice of gesture training through analyzing presentation videos. Manually checking and exploring gesture usage in the presentation videos is often tedious and time-consuming. There lacks an efficient method to help users conduct gesture exploration, which is challenging due to the intrinsically temporal evolution of gestures and their complex correlation to speech content. In this paper, we propose GestureLens, a visual analytics system to facilitate gesture-based and content-based exploration of gesture usage in presentation videos. Specifically, the exploration view enables users to obtain a quick overview of the spatial and temporal distributions of gestures. The dynamic hand movements are firstly aggregated through a heatmap in the gesture space for uncovering spatial patterns, and then decomposed into two mutually perpendicular timelines for revealing temporal patterns. The relation view allows users to explicitly explore the correlation between speech content and gestures by enabling linked analysis and intuitive glyph designs. The video view and dynamic view show the context and overall dynamic movement of the selected gestures, respectively. Two usage scenarios and expert interviews with professional presentation coaches demonstrate the effectiveness and usefulness of GestureLens in facilitating gesture exploration and analysis of presentation videos.

En français or in English? Examining Perceived Social Roles of International Students in Response to Their French and English Speech

Canada’s growing population of international students brings ethnolinguistic diversity and socioeconomic benefits to their host communities. However, students often experience social exclusion and lack of belonging, reporting little communication with local community members for many cultural, ethnic, and religious reasons. The present study, conducted in Montreal, investigated the role of international students’ second language speech in local community members’ perception of the social roles that students can assume in a host society (e.g., friend, neighbour, colleague). Four English- and French-speaking international students’ recordings were presented to 38 francophones, all non-student residents of Montreal, who evaluated the students’ comprehensibility and accentedness and assessed how acceptable these students were in various social roles (e.g., friend, neighbour, colleague) in two languages (French vs. English) and in two situations (making a request vs. making a potentially controversial statement). The students’ ratings were greater when they spoke French than English (regardless of speech content) and when they requested help than when they expressed a controversial statement (regardless of language). Social ratings were generally associated with comprehensibility, not accentedness, where more comprehensible speech was linked to greater perceived acceptability. Findings highlight the importance of sociolinguistic context and language in local residents’ judgments of international students’ social roles.

Confrontational or ‘Statespersonlike’ Style? Examining Finnish and French Presidents’ Public Speeches and Messages, 2000–2020

Presidents can use public speeches for a variety of purposes from rallying support for their initiatives to attacking their opponents or building societal consensus. Contrary to general expectations regarding dual executive systems, this article suggests that presidents in semi-presidential regimes can benefit politically and in terms of popularity from ‘statespersonlike’ behaviour, and thus, they should generally refrain from negative and contentious statements. Examining the tone and content of formal speeches and informal messages of Finnish and French presidents from 2000 to 2020, the study shows that under varying constitutional frameworks and general practical expectations, the presidents in both countries by and large employ a ‘statespersonlike’ style in their speeches. Presidential speeches primarily express a positive tone, and they contain few references to other state institutions and economy and other governmental domains. Instead, presidents typically seek to portray themselves as guardians of national interest, with foreign policy and national unity emphasized in the speeches.

How Does The Bilingualism Improve the Student English Ability: a Discourse Case Study on Higher Education

In this 4.0 era, the use of English as a mean of communications playing a vital role to work in many parts such as technology, education, communication and many others. It is true that not all people can communicate in English fluently and clearly. Luckily, some people or students especially, use bilingualism to study or learn English. Many of them think by using bilingualism they still can interact or communicate by using English as a mean of interaction. Focus of this research was to find out how the bilingualism improve the English student ability. This research used a descriptive qualitative research. Data were collected by using questionnaire and interview guideline. Data were analyzed by using the content analysis and the theory used was proposed by Hoffman 1991. Population of the research was all of students of English for mathematics consisting 35 students. This research used total sampling for instrument using questionnaire and used purposive sampling for the semi-structure interview. The research shows that bilingualism occur in the classroom interaction such as talking about particular topic, quoting somebody else, being emphatic about something, interjection, repetition used for clarification, expressing a group of identity and intention of clarifying the speech content for interlocutor. From the analysis, it is seen that mostly bilingualism used by the students to study about particular topics such as term, theme and content. The least used by student is on the intention of clarifying the speech content for interlocutor because the students always think that what the people say are true and they don’t know how to express the clarification in English. However, for the English learning, bilingualism has already helped both the lecturers and students to interact and communicate about various topics. It is a good start to introduce English as the foreign language In Indonesian Higher Education where English is still considered as the foreign language.

A high-performance speech BioHashing retrieval algorithm based on audio segmentation

As one of the research hotspots in the field of speech recognition, content-based speech retrieval algorithms can detect speech information with the same content features, which improves computer intelligence while reducing labor costs, and thus have been widely used. Although most of the current speech content retrieval algorithms can guarantee excellent retrieval performance for small-scale speech retrieval work, the performance of the above algorithms is greatly reduced under the constraints of large speech data storage space and high content redundancy. In order to solve the above problems, a high-performance speech BioHashing retrieval algorithm based on audio segmentation is proposed in this paper. The algorithm is divided into an offline pre-processing phase and an online retrieval phase, The offline pre-processing stage converts the speech data into BioHashing sequences with speech content characteristics. In this process, first of all, the Power-Normalized Cepstral Coefficients (PNCC) features of the speech data are extracted and biometric templates with single mapping keys are constructed according to the PNCC features, obtaining BioHashing sequences. Then, slice the original speeches into short-time audio segments according to the proposed audio segmentation algorithm, and the hash reconstruction operation is performed on the BioHashing sequences to obtain the reconstructed Hashing sequences for online retrieval. The online search phase responds to the users' query requests, just find the hash index that matches the query hash sequence from the BioHashing index table, and will the standardized editing distance (SED) to the closest 1 value corresponding to the hash index as the retrieval result back to the user. The experimental results show that the reconstructed hash sequences obtained after removing the silent redundant segments have better robustness and discrimination. Moreover, the algorithm achieves 100% retrieval accuracy for the original speech clips, and the average retrieval time is only 0.0157 s, which shows that the algorithm has good retrieval performance and can meet the needs of speech retrieval in various environments.

Voice Orientation Recognition: New Paradigm of Speech-Based Human-Computer Interaction

As one of the most preferred forms of Human-Computer Interaction (HCI) nowadays, speech-based HCI enables people to communicate verbally with machines, leveraging technologies such as speech recognition and speech synthesis. Current paradigm of speech-based HCI focus on the content of speech only, failing to comprehend deeper pointing information in voice interaction. In particular, when encountering scenarios with multiple smart voice devices around, if people intend to interact with a certain device, the lack of extra pointing information (like the role played by the direction of eye gaze) would cause unintended response from the other devices, resulting in poor interaction experience during HCI. Hence, an interesting problem is: Is it possible for the devices to be aware of the orientation of human voice with only the acoustic speech signals? There is little research studying this topic, except for very a few primary works with much room for improvement. The main challenge of this study lies in capturing the concealed orientation information embedded within the speech signal, while simultaneously maintaining the scheme’s practicality and high precision. In this paper, we propose Oriennet, for identifying the orientation of human voice. With a series of features intentionally designed in view of the indoor voice propagation model and mouth radiation pattern, as well as the application of attention mechanism, Oriennet achieve 95% accuracy in terms of judging whether people are facing the device or not. Even for the fine-grained task of classifying people’s specific orientation from 8 different directions, our work achieved an accuracy of 74%, far outperforming the existed works. We have validated the robustness of Oriennet under various conditions (noisy environment; different people, rooms, languages, locations; fewer microphones), demonstrating its promising applicability in real-life scenarios.

Incorporating automatic speech recognition methods into the transcription of police-suspect interviews: factors affecting automatic performance

IntroductionIn England and Wales, transcripts of police-suspect interviews are often admitted as evidence in courts of law. Orthographic transcription is a time-consuming process and is usually carried out by untrained transcribers, resulting in records that contain summaries of large sections of the interview and paraphrased speech. The omission or inaccurate representation of important speech content could have serious consequences in a court of law. It is therefore clear that investigation into better solutions for police-interview transcription is required. This paper explores the possibility of incorporating automatic speech recognition (ASR) methods into the transcription process, with the goal of producing verbatim transcripts without sacrificing police time and money. We consider the potential viability of automatic transcripts as a “first” draft that would be manually corrected by police transcribers. The study additionally investigates the effects of audio quality, regional accent, and the ASR system used, as well as the types and magnitude of errors produced and their implications in the context of police-suspect interview transcripts.MethodsSpeech data was extracted from two forensically-relevant corpora, with speakers of two accents of British English: Standard Southern British English and West Yorkshire English (a non-standard regional variety). Both a high quality and degraded version of each file was transcribed using three commercially available ASR systems: Amazon, Google, and Rev.ResultsSystem performance varied depending on the ASR system and the audio quality, and while regional accent was not found to significantly predict word error rate, the distribution of errors varied substantially across the accents, with more potentially damaging errors produced for speakers of West Yorkshire English.DiscussionThe low word error rates and easily identifiable errors produced by Amazon suggest that the incorporation of ASR into the transcription of police-suspect interviews could be viable, though more work is required to investigate the effects of other contextual factors, such as multiple speakers and different types of background noise.

Voice pitch is negatively associated with sociosexual behavior in males but not in females.

Acoustic cues play a major role in social interactions in many animal species. In addition to the semantic contents of human speech, voice attributes - e.g., voice pitch, formant position, formant dispersion, etc. - have been proposed to provide critical information for the assessment of potential rivals and mates. However, prior studies exploring the association of acoustic attributes with reproductive success, or some of its proxies, have produced mixed results. Here, we investigate whether the mean fundamental frequency (F0), formant position (Pf), and formant dispersion (Df) - dimorphic attributes of the human voice - are related to sociosexuality, as measured by the Revised Sociosexual Orientation Inventory (SOI-R) - a trait also known to exhibit sex differences - in a sample of native Spanish-speaking students (101 males, 147 females). Analyses showed a significant negative correlation between F0 and sociosexual behavior, and between Pf and sociosexual desire in males but not in females. These correlations remained significant after correcting for false discovery rate (FDR) and controlling for age, a potential confounding variable. Our results are consistent with a role of F0 and Pf serving as cues in the mating domain in males but not in females. Alternatively, the association of voice attributes and sociosexual orientation might stem from the parallel effect of male sex hormones both on the male brain and the anatomical structures involved in voice production.

ЛЕКСИКО-СЕМАНТИЧНІ ОСОБЛИВОСТІ МОВИ УКРАЇНСЬКИХ ІНТЕРНЕТ-МЕДІА: РЕЛІГІЙНИЙ ВИМІР

<p><strong><em>The purpose</em></strong><em> </em><em>of the article is to determine the lexical and semantic features of messages on religious theme, the title complex in the Ukrainian segment of the Internet.</em></p><p><strong><em>Research methodology. </em></strong><em>According to the purpose of the research, a number of such methods was used: observation (to collect primary empirical materials), inductive-deductive (to define and describe the peculiarities of messages on religious topics and title complex); comparison, analysis, generalization.</em></p><p><strong><em>Results. </em></strong><em>The analysis has showed that messages of religious theme on media resources presented on the Internet are most often related to the life of Orthodox denominations (OCU and UOC MP) and the Ukrainian Greek-Catholic Church (UGCC). The title complex and the texts of messages often include verbatives with the same meaning of speech and those close to them: verbs of reporting and transmission of information (speak up, to speak, to say); verbs to indicate the content of speech (to tell, to explain); verbs of emotional attitude and evaluation (to condemn, to blame); verbs of speech influence (to call, to appeal). Most often, verbs of speech have a neutral meaning and are devoided of expressive coloring. Sometimes phraseme (idiom) and metaphors are used in the title complex. The use of stylistically elements and tropes is more characteristic of analytical genres: allusion, paronyms, oxymorons, occasionalisms. Metonymy prevails in informational genres. Expressive means of hate speech in the title complex are also not uncommon, in particular nicknames, slang, jargon, vulgar lexis. Publications also actively use the syntactic opportunities of the language, which are typical for online-resources, in particular, sentences in which the second part after the colon does not explain the meaning of the first one, but includes «hook phrase», the purpose of which is to «force» the reader to follow the hyperlink in order to learn more («what happened»; «explained the details»).<strong></strong></em></p><p><strong><em>Novelty </em></strong><em>is that the lexical and semantic features of messages on religious theme in the Ukrainian segment of the Internet have been determined.</em></p><p><strong><em>Practical significance.</em></strong><em> The results of the research can be used in a comprehensive investigation of religious journalism in Ukraine and as educational material in the training of future journalists.</em></p><p><strong><em>Key words:</em></strong><em> religious theme, title complex, verbs of speech,</em><em> </em><em>lexical and semantic groups, hate speech.</em></p>

MALip: Modal Amplification Lipreading based on reconstructed audio features

Lipreading is a type of human–computer interaction based on visual information. From the perspective of pronunciation principle, lip is only one of the vocal tract organs, and it is difficult to express the complete pronunciation process. It is very challenging to recognize speech contents solely through lip movements. As the shape of the vocal tract determines the final sound during the pronunciation process, we propose to improve the accuracy of lipreading by concurrently exploiting the lip features and reconstructed vocal tract features, and we call our method Modal Amplification Lipreading (MALip). We extend the U-Net model to learn the vocal tract features from Mel-Spectrogram features extracted from audio. Our model aims to reduce the computation complexity while ensuring the extended vocal tract features to have a good quality. We also introduce techniques to ensure the vocal tract features to be effective without being compromised by interference from noise or invalid audio. In addition, to facilitate the study with the incorporation of vocal tract features, we recorded a maximum sentence-level Chinese dataset ICSLR in an experimental environment, and verify the effectiveness of reconstructed audio features in improving the lip recognition accuracy for the first time. Through extensive experiments on ICSLR and publicly natural sentence dataset CMLR, we demonstrate the effectiveness of our MALip method compared with state-of-the-art counterparts.