Real-time Captioning Research Articles

How People Living With Amyotrophic Lateral Sclerosis Use Personalized Automatic Speech Recognition Technology to Support Communication.

Amyotrophic lateral sclerosis (ALS) is a progressive, ultimately fatal disease causing progressive muscular weakness. Most people living with ALS (plwALS) experience dysarthria, eventually becoming unable to communicate using natural speech. Many wish to use speech for as long as possible. Personalized automated speech recognition (ASR) model technology, such as Google's Project Relate, is argued to better recognize speech with dysarthria, supporting maintenance of understanding through real-time captioning. The objectives of this study are how plwALS and communication partners use Relate in everyday conversation over a period of up to 12 months and how it may change with any decline in speech over time. This study videoed interactions between three plwALS and communication partners. We assessed ASR caption accuracy and how well they preserved meaning. Conversation analysis was used to identify participants' own organizational practices in the accomplishment of interaction. Thematic analysis was used to understand better the participants' experiences of using ASR captions. All plwALS reported lower-than-expected ASR accuracy when used in conversation and felt ASR captioning was only useful in certain contexts. All participants liked the concept of live captioning and were hopeful that future improvements to ASR accuracy may support their communication in everyday life. Training is needed on best practices for customization and practical use of ASR technology and for the limitations of ASR in conversational settings. Support is needed for those less confident with technology and to reduce misplaced allocation of ownership of captioning errors, risking negative effects on psychological well-being.

Deaf/hard of hearing students' experiences with higher education's real-time captioning services.

Real-time captions appear to be an effective tool in assisting deaf and hard of hearing (DHH) college students' access information and communication in certain classroom settings. However, there is limited knowledge of DHH students' direct experiences with real-time captioning services. In this study, we gathered narratives from 15 DHH college students across the United States about their experiences with real-time captioning services in college. We analyzed the stories using thematic narrative analysis and uncovered 4 types that students told about their experiences. The story types were (a) stories of overcoming obstacles, (b) stories of resignation, (c) pragmatic stories, and (d) stories of personal connection. These story types reveal that although many students eventually experience effective communication access through real-time captioning services, they can initially struggle to overcome barriers to using the services successfully. Making time and space to listen to DHH students' narratives can teach educators and professionals how to support these students and resolve barriers before they arise.

Intellectual Disability and Technology: An Artificial Intelligence Perspective and Framework

Intellectual disability (ID) poses formidable challenges to individuals in acquiring knowledge and performing daily activities. This condition often stems from a variety of factors, such as congenital birth defects, genetic conditions, or acquired injuries, profoundly impacting crucial aspects like speech, mobility, social understanding, and adaptive skills. To address these challenges effectively, early intervention and specialized education play pivotal roles in providing children with intellectual disabilities the necessary support for their development. In this context, Artificial intelligence (AI) emerges as a highly promising tool capable of facilitating effective communication for individuals with intellectual disabilities. AI technology offers personalized communication support, aids in early diagnosis, and provides configurable therapies tailored to the needs of those with intellectual disabilities. Moreover, AI’s contributions extend to enhancing the overall well-being of these individuals, empowering them to achieve greater independence and improved quality of life. Various treatment and support services, such as early intervention programs, special education, and vocational programs, play vital roles in fostering the growth and fulfillment of those with intellectual disabilities, enabling them to thrive in the society. The integration of AI technologies holds tremendous potential to further augment the lives of individuals with disabilities. AI-based applications, including real-time captioning, sign language translation, robotic assistance, virtual reality, and brain–computer interfaces, are transformative tools that promote inclusivity and independence for individuals with intellectual disabilities, paving the way toward a more accessible and empowering future.

Online Presentations with PowerPoint Present Live Real-Time Automated Captions and Subtitles: Perceptions of Faculty and Administrators

Captioning of recorded videos is beneficial to many and a matter of compliance with accessibility regulations and guidelines. Like recorded captions, real-time captions can also be means to implement the Universal Design for Learning checkpoint to offer text-based alternatives to auditory information. A cost-effective solution to implement the checkpoint for live online presentations is to use speech recognition technologies to generate automated captions. In particular, Microsoft PowerPoint Present Live (MSPL) is an application that can be used to present with real-time automated captions and subtitles in multiple languages, allowing individuals to follow the presentation in their preferred language. The purpose of this study was to identify challenges that participants could encounter when using the MSPL feature of real-time automated captions/subtitles, and to determine what they describe as potential uses, challenges, and benefits of the feature. Participants were full-time faculty and administrators with a faculty appointment in a higher education institution. Data from five native English speakers and five native Spanish speakers were analyzed. Activities of remote usability testing and interviews were conducted to collect data. Overall, participants did not encounter challenges that they could not overcome and described MSPL as an easy-to-use and useful tool to present with captions/subtitles for teaching or training and to reach English and Spanish-speaking audiences. The themes that emerged as potential challenges were training, distraction, and technology. Findings are discussed and further research is recommended.

Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning

Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on semi-structured interviews and design probe activities we conducted with 15 DHH participants to understand their use of existing real-time captioning services and future design preferences for both in-person and remote small-group communication. We found that our participants' experiences of captioned small-group conversations are shaped by social, environmental, and technical considerations (e.g., interlocutors' pre-established relationships, the type of captioning displays available, and how far captions lag behind speech). When considering future captioning tools, participants were interested in greater feedback on non-speech elements of conversation (e.g., speaker identity, speech rate, volume) both for their personal use and to guide hearing interlocutors toward more accessible communication. We contribute a qualitative account of DHH people's real-time captioning experiences during small-group conversation and future design considerations to better support the groups being captioned, both in person and online.?

Preview the Products on Display at Display Week 2021

Preview the Products on Display at Display Week 2021

Optimizing Communication in Schools and Other Settings During COVID-19

Optimizing Communication in Schools and Other Settings During COVID-19

Attention-Sharing Initiative of Multimodal Processing in Simultaneous Interpreting

This study sets out to describe simultaneous interpreters' attention-sharing initiatives when exposed under input from both videotaped speech recording and real-time transcriptions. Separation of mental energy in acquiring visual input accords with the human brain's statistic optimization principle where the same property of an object is presented through diverse fashions. In examining professional interpreters' initiatives, the authors invited five professional English-Chinese conference interpreters to simultaneously interpret a videotaped speech with real-time captions generated by speech recognition engine while meanwhile monitoring their eye movements. The results indicate the professional interpreters' preferences in referring to visually presented captions along with the speaker's facial expressions, where low-frequency words, proper names, and numbers gained greater attention than words with higher frequency. This phenomenon might be explained by the working memory theory in which the central executive enables redundancy gains retrieved from dual-channel information.

Crowdsourced real-time captioning of sign language by deaf and hard-of-hearing people

PurposeThe purpose of this paper is to explore the issues on how to achieve crowdsourced real-time captioning of sign language by deaf and hard-of-hearing (DHH) people, such that how a system structure should be designed, how a continuous task of sign language captioning should be divided into microtasks and how many DHH people are required to maintain a high-quality real-time captioning.Design/methodology/approachThe authors first propose a system structure, including the new design of worker roles, task division and task assignment. Then, based on an implemented prototype, the authors analyze the necessary setting for achieving a crowdsourced real-time captioning of sign language, test the feasibility of the proposed system and explore its robustness and improvability through four experiments.FindingsThe results of Experiment 1 have revealed the optimal method for task division, the necessary minimum number of groups and the necessary minimum number of workers in a group. The results of Experiment 2 have verified the feasibility of the crowdsourced real-time captioning of sign language by DHH people. The results of Experiment 3 and Experiment 4 have shown the robustness and improvability of the captioning system.Originality/valueAlthough some crowdsourcing-based systems have been developed for the captioning of voice to text, the authors intend to resolve the issues on the captioning of sign language to text, for which the existing approaches do not work well due to the unique properties of sign language. Moreover, DHH people are generally considered as the ones who receive support from others, but our proposal helps them become the ones who offer support to others.

Real-Time Captioning for Improving Informed Consent: Patient and Physician Benefits.

New methods are needed to improve physicians' skill in communicating information and to enhance patients' ability to recall that information. We evaluated a real-time speech-to-text captioning system that simultaneously provided a speech-to-text record for both patient and anesthesiologist. The goals of the study were to assess hearing-impaired patients' recall of an informed consent discussion about regional anesthesia using real-time captioning and to determine whether the physicians found the system useful for monitoring their own performance. We recorded 2 simulated informed consent encounters with hearing-impaired older adults, in which physicians described regional anesthetic procedures. The conversations were conducted with and without real-time captioning. Subsequently, the patient participants, who wore their hearing aids throughout, were tested on the material presented, and video recordings of the encounters were analyzed to determine how effectively physicians communicated with and without the captioning system. The anesthesiology residents provided similar information to the patient participants regardless of whether the real-time captioning system was used. Although the patients retained relatively few details regardless of the informed consent discussion, they could recall significantly more of the key points when provided with real-time captioning. Real-time speech-to-text captioning improved recall in hearing-impaired patients and proved useful for determining the information provided during an informed consent encounter. Real-time speech-to-text captioning could provide a method for assessing physicians' communication that could be used both for self-assessment and as an evaluative approach to training communication skills in practice settings.

Admission of and Service Provision for Deaf Students at a University of Technology

Research reveals that there is a lack of research on Deaf students in tertiary institutions in South Africa and that there are not many tertiary institutions which enrol Deaf students and provide them with sign language interpreters. While a considerable amount of research has been done on what constitutes Deaf students’ barriers to gaining access to institutions of higher learning, not as much is known about the potential solutions to these problems. In addition, there is a lack of inquiry into the value of support services such as interpreting, note taking, real-time captioning and tutoring, particularly with regard to their impact on academic achievement. The purpose of this paper is to explore ways in which the quality of education for deaf students can be enhanced through access and support services in one institution of higher education. DOI: 10.5901/mjss.2014.v5n10p370

Avoiding communication barriers in the classroom: the APEINTA project

Education is a fundamental human right, however unfortunately not everybody has the same learning opportunities. For instance, if a student has hearing impairments, s/he could face communications barriers in the classroom, which could affect his/her learning process. APEINTA is a Spanish educational project that aims for inclusive education for all. This project proposes two main accessible initiatives: (1) real-time captioning and text-to-speech (TTS) services in the classroom and (2) accessible Web-learning platform out of the classroom with accessible digital resources. This paper presents the inclusive initiatives of APEINTA. Also an evaluation of the into-the-classroom initiative (real-time captioning and TTS services) is presented. This evaluation has been conducted during a regular undergraduate course at a university and during a seminar at an integration school for deaf children. Forty-five hearing students, 1 foreign student, 3 experts in captioning, usability and accessibility, and 20 students with hearing impairments evaluated these services in the classroom. Evaluation results show that these initiatives are adequate to be used in the classroom and that students are satisfied with them.

Observations of Real-Time Captioning in the Elementary English Language Learner Classroom

This qualitative study reviews the effects of real-time captioning on vocabulary acquisition of a novice elementary English language learner (ELL). Triangulation of data was completed through review of teacher journaling, captioning transcripts, and student pretest and posttest assessments. Results illuminate improved retention and usage of English during the captioned class sessions. Findings are discussed through the lens of the Hawthorne effect and social development theory. The researcher explores implications for ELL teachers, school administrators, and institutions of higher education.

Accessibility Evaluation of Classroom Captions

Real-time captioning enables deaf and hard of hearing (DHH) people to follow classroom lectures and other aural speech by converting it into visual text with less than a five second delay. Keeping the delay short allows end-users to follow and participate in conversations. This article focuses on the fundamental problem that makes real-time captioning difficult: sequential keyboard typing is much slower than speaking. We first surveyed the audio characteristics of 240 one-hour-long captioned lectures on YouTube, such as speed and duration of speaking bursts. We then analyzed how these characteristics impact caption generation and readability, considering specifically our human-powered collaborative captioning approach. We note that most of these characteristics are also present in more general domains. For our caption comparison evaluation, we transcribed a classroom lecture in real-time using all three captioning approaches. We recruited 48 participants (24 DHH) to watch these classroom transcripts in an eye-tracking laboratory. We presented these captions in a randomized, balanced order. We show that both hearing and DHH participants preferred and followed collaborative captions better than those generated by automatic speech recognition (ASR) or professionals due to the more consistent flow of the resulting captions. These results show the potential to reliably capture speech even during sudden bursts of speed, as well as for generating “enhanced” captions, unlike other human-powered captioning approaches.

聴覚障害者が読みやすい英語音声認識字幕呈示の改行条件に関する研究

本研究は、聴覚障害者にとって読みやすい英語の音声認識字幕の改行について知見を得るための研究である。日本人の先天性重度聴覚障害者と聴者を対象とし、理解・生成の意味単位であるチャンクに基づいて1行あたりの平均単語数が異なる5つの改行条件で字幕文章を呈示し、読みやすさの主観的評価と字幕読解時の眼球運動測定を行った。その結果、聴覚障害者・聴者ともに、ピリオドやカンマで改行する呈示文で読みやすさの評価が高かった。また、聴覚障害対象者において、サッカード距離と平均注視時間が長く逆行回数の少ない対象者では、ピリオドでの改行条件の評価が高く、サッカード距離と平均注視時間が短く逆行回数の多い対象者では、カンマとピリオドでの改行条件の評価が高かった。以上の結果から1）読みのストラテジーの違いによって読みやすいと感じる改行位置が異なる、2）読み返しのしやすさが読みやすさの目安のひとつになることが示唆された。

Using speech recognition for real-time captioning and lecture transcription in the classroom

Speech recognition (SR) technologies were evaluated in different classroom environments to assist students to automatically convert oral lectures into text. Two distinct methods of SR-mediated lecture acquisition (SR-mLA), real-time captioning (RTC) and postlecture transcription (PLT), were evaluated in situ life and social sciences lecture courses employing typical classroom equipment. Both methods were compared according to technical feasibility and reliability of classroom implementation, instructors' experiences, word recognition accuracy, and student class performance. RTC provided near-instantaneous display of the instructor's speech for students during class. PLT employed a user-independent SR algorithm to optimally generate multimedia class notes with synchronized lecture transcripts, instructor audio, and class PowerPoint slides for students to access online after class. PLT resulted in greater word recognition accuracy than RTC. During a science course, students were more likely to take optional online quizzes and received higher quiz scores with PLT than when multimedia class notes were unavailable. Overall class grades were also higher when multimedia class notes were available. The potential benefits of SR-mLA for students who have difficulty taking notes accurately and independently were discussed, particularly for nonnative English speakers and students with disabilities. Field-tested best practices for optimizing SR accuracy for both SR-mLA methods were outlined.

Deafness Among Physicians and Trainees

To describe the characteristics of and accommodations used by the deaf and hard-of-hearing (DHoH) physician and trainee population and examine whether these individuals are more likely to care for DHoH patients. Multipronged snowball sampling identified 86 potential DHoH physician and trainee participants. In July to September 2010, a Web-based survey investigated accommodations used by survey respondents. The authors analyzed participants' demographics, accommodation and career satisfaction, sense of institutional support, likelihood of recommending medicine as a career, and current/anticipated DHoH patient population size. The response rate was 65% (56 respondents; 31 trainees and 25 practicing physicians). Modified stethoscopes were the most frequently used accommodation (n = 50; 89%); other accommodations included auditory equipment, note-taking, computer-assisted real-time captioning, signed interpretation, and oral interpretation. Most respondents reported that their accommodations met their needs well, although 2 spent up to 10 hours weekly arranging accommodations. Of 25 physicians, 17 reported primary care specialties; 7 of 31 trainees planned to enter primary care specialties. Over 20% of trainees anticipated working with DHoH patients, whereas physicians on average spent 10% of their time with DHoH patients. Physicians' accommodation satisfaction was positively associated with career satisfaction and recommending medicine as a career. DHoH physicians and trainees seemed satisfied with frequent, multimodal accommodations from employers and educators. These results may assist organizations in planning accommodation provisions. Because DHoH physicians and trainees seem interested in primary care and serving DHoH patients, recruiting and training DHoH physicians has implications for the care of this underserved population.

5-2 デジタル放送に対応したリアルタイム字幕を適切なタイミングで表示できる字幕制作装置と受信装置の研究開発(第5部門 放送方式,現業,無線・光伝送)

5-2 デジタル放送に対応したリアルタイム字幕を適切なタイミングで表示できる字幕制作装置と受信装置の研究開発(第5部門 放送方式,現業,無線・光伝送)

Improving the experience of deaf students in higher education

This article identifies and critically analyses the difficulties faced by deaf students studying in higher education settings and the potential support that they require. Deaf students face significant barriers in gaining access to information and learning in the classroom. At the same time, the number of such students attending universities and colleges has increased dramatically over the past 20 years. Little known about the potential solutions to these problems, and there is a dearth of research on the effectiveness of support services made available, including interpreting, note-taking, real-time captioning (a written rendering of the dialogue spoken, in the same language) and tutoring, particularly when attempting to take into account the impact of such support on academic achievement. The role of motivation is discussed alongside barriers and levers to education in this group. The role of the lecturer in supporting these students is explored.

Proposal of Mobile Type Remote-captioning System for Deaf or Hard of HearingPeople and Evaluation by Captionists

Deaf or hard-of-hearing people need support services such as real-time captioning. It is fairly common for captionists to be unable to perform captioning services. In order to address this problem, we proposed a Mobile-type Remote-captioning System. The main characteristic of our system is to use two functions of one mobile phone (e.g. the iPhone 3G/3GS, Apple Computer, Inc.) at the same time. The two functions are to display the captions input by the captionists (data communications) and to call the captionists to transmit the speaker's speech to the captionists (telephone call). The purpose of our research was to make clear the differences with the existing system, and to discover the advantages and faults of our system. In this study, we investigated these issues using a questionnaire for captionists who had used our system. As a result, we found that our system is suitable for a situation in which the speaker and the deaf or hard-of-hearing person are communicating while walking together (e.g. excursions, plant tours), whether indoors or outdoors. In addition, it reduces the stress that can be caused to deaf or hard-of-hearing children by having adults (captionists) other than teachers in the classroom.