Real-time Language Research Articles

Machine Learning-Based Audio Interface Model for Sign Language Recognition

Due to the fact that most offices and educational institutions now operate from home, the work-from-home and study-from-home cultures have made it difficult to interact with persons who are deaf or hard of hearing. These people communicate within their society using sign language, which is not widely understood by others. Most of the time, as a result of this, they miss out on the opportunity to express their point in front of every one since they are ignored/passed over without receiving the necessary attention. In real-time, having an independent translator that can process photos and interpret signs quickly at the speed of streaming images is critical. We'll utilize TensorFlow Object Detection and Python to bridge the gap by creating an end-to-end bespoke object detection model that not only translates sign language in real time but also speaks it to others.

The emergence of bifurcated structure in children's language.

Human language is unique among animal communication systems, in part because of its dual patterning in which meaningless phonological units combine to form meaningful words (phonological structure) and words combine to form sentences (lexicosyntactic structure). Although dual patterning is well recognized, its emergence in language development has been scarcely investigated. Chief among questions still unanswered is the extent to which development of these separate structures is independent or interdependent, and what supports acquisition of each level of structure. We explored these questions by examining growth of lexicosyntactic and phonological structure in children with normal hearing (n = 49) and children with hearing loss who use cochlear implants (n = 56). Multiple measures of each kind of structure were collected at 2-year intervals (kindergarten through eighth grade), and used to construct latent scores for each type of structure. Growth curve analysis assessed (a) the relative independence of development for each level of structure; (b) interactions between these two levels of structure in real-time language processing; and (c) contributions to growth of each level of structure made by auditory input, socioeconomic status (as proxy for linguistic experience), and speech motor control. Findings suggested that phonological and lexicosyntactic structure develop largely independently. Auditory input, socioeconomic status, and speech motor control help shape these language structures, with the last two factors exerting stronger effects for children with cochlear implants. Only for children with cochlear implants were interdependencies in real-time processing observed, reflecting compensatory mechanisms likely present to help them handle the disproportionately large phonological deficit they exhibit. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A cognitive modeling approach to learning and using reference biases in language.

During real-time language processing, people rely on linguistic and non-linguistic biases to anticipate upcoming linguistic input. One of these linguistic biases is known as the implicit causality bias, wherein language users anticipate that certain entities will be rementioned in the discourse based on the entity's particular role in an expressed causal event. For example, when language users encounter a sentence like "Elizabeth congratulated Tina…" during real-time language processing, they seemingly anticipate that the discourse will continue about Tina, the object referent, rather than Elizabeth, the subject referent. However, it is often unclear how these reference biases are acquired and how exactly they get used during real-time language processing. In order to investigate these questions, we developed a reference learning model within the PRIMs cognitive architecture that simulated the process of predicting upcoming discourse referents and their linguistic forms. Crucially, across the linguistic input the model was presented with, there were asymmetries with respect to how the discourse continued. By utilizing the learning mechanisms of the PRIMs architecture, the model was able to optimize its predictions, ultimately leading to biased model behavior. More specifically, following subject-biased implicit causality verbs the model was more likely to predict that the discourse would continue about the subject referent, whereas following object-biased implicit causality verbs the model was more likely to predict that the discourse would continue about the object referent. In a similar fashion, the model was more likely to predict that subject referent continuations would be in the form of a pronoun, whereas object referent continuations would be in the form of a proper name. These learned biases were also shown to generalize to novel contexts in which either the verb or the subject and object referents were new. The results of the present study demonstrate that seemingly complex linguistic behavior can be explained by cognitively plausible domain-general learning mechanisms. This study has implications for psycholinguistic accounts of predictive language processing and language learning, as well as for theories of implicit causality and reference processing.

Associations between early efficiency in language processing and language and cognitive outcomes in children born full term and preterm: similarities and differences

ABSTRACT Associations between children’s early language processing efficiency and later verbal and non-verbal outcomes shed light on the extent to which early information processing skills support later learning across different domains of function. Examining whether the strengths of associations are similar in typically developing and at-risk populations provides an additional lens into the varying routes to learning that children may take across development. In this follow-up study, children born full-term (FT, n = 49) and preterm (PT, n = 45, ≤32 weeks gestational age, birth weight <1800 g) were assessed in the Looking While Listening (LWL) task at 18 months (corrected for degree of prematurity in PT group). This eye-tracking task assesses efficiency of real-time spoken language comprehension as accuracy and speed (RT) of processing. At 4 ½ years, children were assessed on standardized tests of receptive vocabulary, expressive language, and non-verbal IQ. Language processing efficiency was associated with both language outcomes (r 2-change: 7.0–19.7%, p < 0.01), after covariates. Birth group did not moderate these effects, suggesting similar mechanisms of learning in these domains for PT and FT children. However, birth group moderated the association between speed and non-verbal IQ (r 2-change: 4.5%, p < 0.05), such that an association was found in the PT but not the FT group. This finding suggests that information processing skills reflected in efficiency of real-time language processing may be recruited to support learning in a broader range of verbal and non-verbal domains in the PT compared to the FT group.

Moving visual world experiments online? A web-based replication of Dijkgraaf, Hartsuiker, and Duyck (2017) using PCIbex and WebGazer.js.

The visual world paradigm is one of the most influential paradigms to study real-time language processing. The present study tested whether visual world studies can be moved online, using PCIbex software (Zehr & Schwarz, 2018) and the WebGazer.js algorithm (Papoutsaki et al., 2016) to collect eye-movement data. Experiment 1 was a fixation task in which the participants looked at a fixation cross in multiple positions on the computer screen. Experiment 2 was a web-based replication of a visual world experiment by Dijkgraaf et al. (2017). Firstly, both experiments revealed that the spatial accuracy of the data allowed us to distinguish looks across the four quadrants of the computer screen. This suggest that the spatial resolution of WebGazer.js is fine-grained enough for most visual world experiments (which typically involve a two-by-two quadrant-based set-up of the visual display). Secondly, both experiments revealed a delay of roughly 300 ms in the time course of the eye movements, possibly caused by the internal processing speed of the browser or WebGazer.js. This delay can be problematic in studying questions that require a fine-grained temporal resolution and requires further investigation.

Parental tuning of language input to autistic and nonspectrum children.

Caregivers’ language input supports children’s language development, and it is often tuned to the child’s current level of skill. Evidence suggests that parental input is tuned to accommodate children’s expressive language levels, but accommodation to receptive language abilities is less understood. In particular, little is known about parental sensitivity to children’s abilities to process language in real time. Compared to nonspectrum children, children on the spectrum are slower to process language. In this study, we ask: Do parents of autistic children and those of nonspectrum children tune their language input to accommodate children’s different language processing abilities? Children with and without a diagnosis of autism (ages 2–6 years, N = 35) and their parents viewed a display of six images, one of which was the target. The parent labeled the target to direct the child’s attention to it. We first examined children’s language processing abilities by assessing their latencies to shift gaze to the labeled referent; from this, we found slower latencies in the autistic group than in the nonspectrum group, in line with previous findings. We then examined features of parents’ language and found that parents in both groups produced similar language, suggesting that parents may not adjust their language input according to children’s speed of language processing. This finding suggests that (1) capturing parental sensitivity to children’s receptive language, and specifically language processing, may enrich our models of individual differences in language input, and (2) future work should investigate if supporting caregivers in tuning their language use according to children’s language processing can improve children’s language outcomes.

Speech or Text to Indian Sign Language Convertor

Abstract: Sign language display is the software of converting the text/speech to animated sign language to meet the demands of the special needs population. The goal of this is to make the communication A more comfortable and to improve both health and productivity .As technology has improved, the usage of computer systems has increased, which means we can implement better ways to meet the needs that can impact the health of special needs population . So, this project could help people improve their communication. Thus, they improve their mental health .With the help of Blender, Python, SQL, and NLP, We have tried to design a process to detect text and subsequently the live speech , which can converted to animated sign language in real time. For the entire process, I used Blender for animation and video processing and SQL and NLP to determine the different words and text to by training the dataset and for text to animation conversion .Therefore, this project could to meet the needs of many people .The number of different countries use different sign language even same countries that are multilingual use a different system which meets the social and cultural standards of their countries. As such there is a wide disparity in many countries for digital sign language But, Software such as our could be helpful to bridge the gap between such discrepancies and be useful not just for acting as a tool for communication but also as tool use to teach . In conclusion , it is very cost effective and widely applicable

INVESTIGATING INDONESIAN EFL TEACHERS’ PERCEPTIONS ON THEIR SELF-EFFICACY

To fully arm Indonesian EFL educationalists with a higher degree of professionalism, tenacity, and optimism in confronting ever-changing ELT landscapes, self-efficacy is one of the pivotal psychological streams that should be strong-inherited within them before plunging into real-time language learning contexts. This present study attempted to investigate the specific perceptions upholding by Indonesian EFL teachers toward their self-efficacy growth amid varied language learning dynamics they constantly endure. This study was run in the fashion of qualitative method to obtain more comprehensive data out of the particular phenomenon experienced by research participants. To fulfill this major research objectivity, the researcher capitalized on 5 open-ended written narrative inquiries to gain some renewable insights from the invited research participants. 2 Indonesian EFL teachers possessing a distinctive length of teaching experiences were invited to take part in answering the aforesaid 5 open-ended written narrative inquiry questions through WhatsApp application. More specifically, these 2 invited Indonesian EFL teachers enrolling in the English Education Master Program of Sanata Dharma University, Yogyakarta. The obtained research results uncovered that the robust establishment of self-efficacy was strongly affected by varied external factors such as a higher degree of learners’ learning endeavor and intensive professional development training addressed by educational institutions, which in turn transforming them into more professional educators. As a suggestive point, all educational parties in this archipelago were strongly advocated to design continual professional development programs heeding more profound attention in corroborating our EFL teachers’ self-efficacy levels for the advancement of ELT venture thoroughly.&#x0D; &#x0D; Keywords: EFL teachers’ perceptions, self-efficacy, narrative inquiry

Real-Time Sign Language Detection using TensorFlow, OpenCV and Python

Abstract: Deaf and hard-of-hearing persons, as well as others who are unable to communicate verbally, utilise sign language to communicate within their communities and with others. Sign languages are a set of preset languages that communicate information using a visual-manual modality. The dilemma of real-time finger-spelling recognition in Sign Language is discussed. We gathered a dataset for identifying 36 distinct gestures (alphabets and numerals) and a dataset for typical hand gestures in ISL created from scratch using webcam images. The system accepts a hand gesture as input and displays the identified character on the monitor screen in real time. This project falls under the category of human-computer interaction (HCI) and tries to recognise multiple alphabets (a-z), digits (0-9) and several typical ISL hand gestures. To apply Transfer learning to the problem, we used a Pre-Trained SSD Mobile net V2 architecture trained on our own dataset. In the vast majority of situations, we constructed a robust model that consistently classifies Sign language. Many studies have been done in the past in this area employing sensors (such as glove sensors) and other image processing techniques (such as edge detection technique, Hough Transform, and so on), but these technologies are quite costly, and many people cannot afford them. During the study, various human-computer interaction approaches for posture recognition were investigated and evaluated. The optimum solution was determined to comprise a set of image processing approaches with Human movement categorization. Without a controlled background and low light, the system can detect chosen Sign Language signs with an accuracy of 70-80%. As a result, we're creating this software to assist such folks because it's free and simple to use. However, aside from a small group of people, not everyone is familiar with sign language, and they may need an interpreter, which may be cumbersome and costly. This research intends to bridge the communication gap by building algorithms that can anticipate alphanumeric hand motions in sign language in real time. The main goal of this research is to create a computer-based intelligent system that will allow deaf persons to interact effectively with others by utilising hand gestures. Keywords: Pre-Trained SSD Mobile net V2, Sign Language , HCI

Implementation of Robotic Navigation Algorithms Using Partial Reconfiguration on Zynq SoC

The purpose for this research work is to design a heterogeneous method for adaptive, task-based computer hardware reconfiguration for adaptive mobile robotic applications. The proposed method uses the dynamic reconfiguration technique to modify the navigation system to meet the requirements of various navigation algorithms for accomplishing an assigned task. In this work, we have proposed an efficient architecture that satisfies the requirements with adequate hardware and software resources by dynamically reconfiguring the system. The ability to change resources allows robots to collaborate more easily, to enhance performance and fault tolerance. The method is endorsed with case studies in which a multiple mobile robots are loaded with bitfile and their behaviors are dynamically modified during runtime. This paper also proposes the hardware implementation of navigation algorithms on Zynq SoC, XC7Z020CLG484-1, using Xilinx Vivado 2017.3 to simulate and synthesize the design. Verilog is a real-time hardware description language, which is used to deploy multiple robots. Software Development Kit (SDK) is used to facilitate the generation of integrated software applications for Xilinx embedded processors. One of the important aspects of this work is adopting an FPGA-based robot in service-based applications for assisting humans in their daily lives.

Speak To Classify Your Notes

Abstract: Over the past few decades, note taking, which was used mostly in school and at university, has been used in everyday life after university as well. Writing can be time-consuming, especially in a fast lecture. Typing takes the least amount of time, so the page can have more information and be reviewed later. It consists of two phases speech to text conversion and classification both plays an important role. Speech-to-text conversion facilitates the integration of people with hearing impairments into oral communication settings. However, the transformation of speech into written language in real time requires specific techniques as it must be very fast and almost 80% correct to be understood. In machine learning, classification is the process of assigning a set of predefined categories to open-ended text. Text classifiers are capable of organizing, structuring, and categorizing pretty much any kind of text. Keywords: Speech To Text, Text Classifiers, Classification, Digital Note-Taking, Machine Learning Technique

Wrong or right? Brain potentials reveal hemispheric asymmetries to semantic relations during word-by-word sentence reading as a function of (fictional) knowledge

How does individual-level variation in experience and knowledge influence neural mechanisms important for real-time language comprehension? We used event-related brain potentials (ERPs) combined with lateralized visual field (VF) presentation of sentence-final words to examine asymmetries in hemispheric processing as individuals who varied in their knowledge of the fictional world of Harry Potter (HP) read sentences about general topics and HP. HP sentence endings were either contextually supported, unrelated anomalies, or semantically related anomalies. Amongst HP experts, we observed contextual support effects with presentation to both left and right VFs, but related anomaly effects only with presentation to the left VF (right hemisphere). Our findings are in line with accounts in which the left hemisphere (LH) activates relatively narrow/specific semantic information and the RH activates a broader range of semantic relations, including those relating to the construction of mental/situation models, as individuals attempt to comprehend sentences, one word at a time. We suggest that RH-biased hemispheric asymmetries in processing related (but linguistically inappropriate) words in written sentences may emerge as a function of each individual's degree of relevant knowledge. We tentatively hypothesize that content experts may optimize hemispheric differences in scope of semantic activation to maximize both precision (in the LH) and flexibility (in the RH) during language comprehension.

Tracking Object-State Representations During Real-Time Language Comprehension by Native and Non-native Speakers of English.

The present “visual world” eye-tracking study examined the time-course of how native and non-native speakers keep track of implied object-state representations during real-time language processing. Fifty-two native speakers of English and 46 non-native speakers with advanced English proficiency joined this study. They heard short stories describing a target object (e.g., an onion) either having undergone a substantial change-of-state (e.g., chop the onion) or a minimal change-of-state (e.g., weigh the onion) while their eye movements toward competing object-states (e.g., a chopped onion vs. an intact onion) and two unrelated distractors were tracked. We found that both groups successfully directed their visual attention toward the end-state of the target object that was implied in the linguistic context. However, neither group showed anticipatory eye movements toward the implied object-state when hearing the critical verb (e.g., “weigh/chop”). Only native English speakers but not non-native speakers showed a bias in visual attention during the determiner (“the”) before the noun (e.g., “onion”). Our results suggested that although native and non-native speakers of English largely overlapped in their time-courses of keeping track of object-state representations during real-time language comprehension, non-native speakers showed a short delay in updating the implied object-state representations.

Comparison of Structural Parsers and Neural Language Models as Surprisal Estimators.

Expectation-based theories of sentence processing posit that processing difficulty is determined by predictability in context. While predictability quantified via surprisal has gained empirical support, this representation-agnostic measure leaves open the question of how to best approximate the human comprehender's latent probability model. This article first describes an incremental left-corner parser that incorporates information about common linguistic abstractions such as syntactic categories, predicate-argument structure, and morphological rules as a computational-level model of sentence processing. The article then evaluates a variety of structural parsers and deep neural language models as cognitive models of sentence processing by comparing the predictive power of their surprisal estimates on self-paced reading, eye-tracking, and fMRI data collected during real-time language processing. The results show that surprisal estimates from the proposed left-corner processing model deliver comparable and often superior fits to self-paced reading and eye-tracking data when compared to those from neural language models trained on much more data. This may suggest that the strong linguistic generalizations made by the proposed processing model may help predict humanlike processing costs that manifest in latency-based measures, even when the amount of training data is limited. Additionally, experiments using Transformer-based language models sharing the same primary architecture and training data show a surprising negative correlation between parameter count and fit to self-paced reading and eye-tracking data. These findings suggest that large-scale neural language models are making weaker generalizations based on patterns of lexical items rather than stronger, more humanlike generalizations based on linguistic structure.

Do Children Use Multi-Word Information in Real-Time Sentence Comprehension?

Meaning in language emerges from multiple words, and children are sensitive to multi-word frequency from infancy. While children successfully use cues from single words to generate linguistic predictions, it is less clear whether and how they use multi-word sequences to guide real-time language processing and whether they form predictions on the basis of multi-word information or pairwise associations. We address these questions in two visual-world eye-tracking experiments with 5- to 8-year-old children. In Experiment 1, we asked whether children generate more robust predictions for the sentence-final object of highly frequent sequences (e.g., "Throw the ball"), compared to less frequent sequences (e.g., "Throw the book"). We further examined if gaze patterns reflect event knowledge or phrasal frequency by comparing the processing of phrases that have the same event structure but differ in multi-word content (e.g., "Brush your teeth" vs. "Brush her teeth"). In the second study, we employed a training paradigm to ask if children are capable of generating predictio.ns from novel multi-word associations while controlling for the overall frequency of the sequences. While the results of Experiment 1 suggested that children primarily relied on event associations to generate real-time predictions, those of Experiment 2 showed that the same children were able to use recurring novel multi-word sequences to generate real-time linguistic predictions. Together, these findings suggest that children can draw on multi-word information to generate linguistic predictions, in a context-dependent fashion, and highlight the need to account for the influence of multi-word sequences in models of language processing.

A Portable Sign Language Collection and Translation Platform with Smart Watches Using a BLSTM-Based Multi-Feature Framework.

Continuous sign language recognition (CSLR) using different types of sensors to precisely recognize sign language in real time is a very challenging but important research direction in sensor technology. Many previous methods are vision-based, with computationally intensive algorithms to process a large number of image/video frames possibly contaminated with noises, which can result in a large translation delay. On the other hand, gesture-based CSLR relying on hand movement data captured on wearable devices may require less computation resources and translation time. Thus, it is more efficient to provide instant translation during real-world communication. However, the insufficient amount of information provided by the wearable sensors often affect the overall performance of this system. To tackle this issue, we propose a bidirectional long short-term memory (BLSTM)-based multi-feature framework for conducting gesture-based CSLR precisely with two smart watches. In this framework, multiple sets of input features are extracted from the collected gesture data to provide a diverse spectrum of valuable information to the underlying BLSTM model for CSLR. To demonstrate the effectiveness of the proposed framework, we test it on an extremely challenging and radically new dataset of Hong Kong sign language (HKSL), in which hand movement data are collected from 6 individual signers for 50 different sentences. The experimental results reveal that the proposed framework attains a much lower word error rate compared with other existing machine learning or deep learning approaches for gesture-based CSLR. Based on this framework, we further propose a portable sign language collection and translation platform, which can simplify the procedure of collecting gesture-based sign language dataset and recognize sign language through smart watch data in real time, in order to break the communication barrier for the sign language users.

Real-time sign language recognition and speech conversion using VGG16

Real-time sign language recognition and speech conversion using VGG16

Composition within and between Languages in the Bilingual Mind: MEG Evidence from Korean/English Bilinguals.

The ability of the human brain to build complex expressions from simpler parts is fascinating, but the ability of the bilingual brain to do so is perhaps even more remarkable. When highly proficient bilinguals converse, they can fluidly switch from one language to another even inside sentences. Thus, they build expressions using words from more than one language. How are bilinguals able to compose words across different languages in real time? While robust evidence has implicated the left anterior temporal lobe (LATL) for the composition of words within one language, we do not know how the LATL, or other regions implicated for composition, operates when the language switches. We also do not know whether prefrontal regions associated with language control are recruited for language switching during composition. We addressed these questions with magnetoencephalography measurements in bilinguals who are fluent in two typologically distant languages, English and Korean. We observed early composition effects in the LATL at ∼200 ms that were unaffected by either language or orthography switching, which was also varied (Hangul vs Roman alphabet). Thus, the combinatory mechanism at 200 ms housed in the anterior temporal cortex appears blind to the language in which its input concepts are expressed. However, in later time windows, language and orthography switching interacted both in regions implicated for composition [LATL, ventromedial prefrontal cortex, left inferior frontal gyrus (LIFG)] as well as in regions associated with language control (ACC, LIFG). This establishes a starting point for understanding how bilingual brains code switch: words are initially combined without consideration of which language they come from, but language switching affects later processing.

Monitoring 21st-Century Real-Time Language Change in Spanish Youth Speech

In recent decades, youth language has become one of the preferred research areas in sociolinguistics, not only because of its non-normative nature but mostly because it is recognized as a catalyst for language change. Since adolescents aspire to create and safeguard an in-group identity, they constantly generate innovative linguistic forms. However, few studies have empirically monitored the speed at which linguistic innovations are introduced into youth language. This study explores the speed and nature of recent language change within Spanish youth language by conducting a corpus analysis in real time. Data of the contemporary CORMA corpus (Corpus Oral de Madrid, compiled between 2016 and 2019) are contrasted with the highly comparable data of the COLAm corpus (Corpus Oral de Lenguaje Adolescente de Madrid, compiled between 2003 and 2007). The study scrutinizes two typical phenomena of youth language, namely the use of intensifiers (e.g., super-, mazo) and vocatives (e.g., tío/tía, chaval/chavala). It is shown that changes occur at a more moderate speed than previously assumed and that the speed of change depends on the linguistic phenomenon under study. Additionally, the data suggest that more neutral forms remain quite stable over time, while the use of more expressive items shrinks or increases faster.

Learning Nondeterministic Real-Time Automata

We present an active learning algorithm named NRTALearning for nondeterministic real-time automata (NRTAs). Real-time automata (RTAs) are a subclass of timed automata with only one clock which resets at each transition. First, we prove the corresponding Myhill-Nerode theorem for real-time languages. Then we show that there exists a unique minimal deterministic real-time automaton (DRTA) recognizing a given real-time language, but the same does not hold for NRTAs. We thus define a special kind of NRTAs, named residual real-time automata (RRTAs), and prove that there exists a minimal RRTA to recognize any given real-time language. This transforms the learning problem of NRTAs to the learning problem of RRTAs. After describing the learning algorithm in detail, we prove its correctness and polynomial complexity. In addition, based on the corresponding Myhill-Nerode theorem, we extend the existing active learning algorithm NL* for nondeterministic finite automata to learn RRTAs. We evaluate and compare the two algorithms on two benchmarks consisting of randomly generated NRTAs and rational regular expressions. The results show that NRTALearning generally performs fewer membership queries and more equivalence queries than the extended NL* algorithm, and the learnt NRTAs have much fewer locations than the corresponding minimal DRTAs. We also conduct a case study using a model of scheduling of final testing of integrated circuits.