Audio Modes Research Articles

Multimodal Hong Kong: a multisensory study of environmental perception of places through sight, sound, and smell

Despite the importance of soundscape and smellscape in intangible cultural heritage, little is known about multisensory interactions. To explore the impact of multiple senses on environmental perception, the present study focused on six stalls at Shek Kip Mei wet market in Hong Kong. In the first part of the study, an online survey collected perceptual ratings of whole, auditory, and visual environmental quality, as well as free-form responses of imaginations of smell. Participants (N = 36) were presented with 12 audiovisual captures where audio was either matched with video or mismatched. ANOVA was used to examine the influence of three independent factors - scenes, modes, and a priori valence groups - on the ratings. The imagined smells were analysed through frequency counts of keywords. The results demonstrate participants' capacity to notice changes in audio modes and to perceive the environment through the recognition of scene context and multisensory interactions. In the second part of the study, not discussed in the present paper, we will extend this experimental method to include smells in a laboratory setting. We discuss the importance of understanding multisensory interactions between sight, sound, and smell for environmental perceptions and Hong Kong's intangible cultural heritage and urban studies.

Multimodal Representations of Rural-Urban Divide on Reality TV: The Case of X-Change

China, as the most populous developing nation globally, may exhibit the greatest urban-rural disparity compared to other developing nations. Perspectives on the rural-urban divide persist despite the significant political, economic, and social upheaval that have occurred in Mainland China over the past century. Drawing on Mackay’s (2015) multimodal legitimation framework, this study examined how urban and rural were constructed and (de)legitimized by a series of multimodal resources, including visual, verbal, and audio modes. Using a critical discourse analysis perspective through a multimodal legitimation approach, this study explored how X-Change portrayed and influenced current Chinese perceptions of reality, with a specific focus on how these programmes depicted China’s urban-rural divide. The findings revealed that both the urban and rural were multimodally constructed and (de)legitimized as a paradox. However, this study also found a reproduction of urban hegemonic discourse via the interplay of various semiotic resources and (de)legitimation strategies, suggesting that the urban very much forms the center of society in the context of contemporary China. These findings contribute to existing scholarship on the rural-urban divide in Chinese Reality Television (RTV).

StreetAware: A High-Resolution Synchronized Multimodal Urban Scene Dataset.

Access to high-quality data is an important barrier in the digital analysis of urban settings, including applications within computer vision and urban design. Diverse forms of data collected from sensors in areas of high activity in the urban environment, particularly at street intersections, are valuable resources for researchers interpreting the dynamics between vehicles, pedestrians, and the built environment. In this paper, we present a high-resolution audio, video, and LiDAR dataset of three urban intersections in Brooklyn, New York, totaling almost 8 unique hours. The data were collected with custom Reconfigurable Environmental Intelligence Platform (REIP) sensors that were designed with the ability to accurately synchronize multiple video and audio inputs. The resulting data are novel in that they are inclusively multimodal, multi-angular, high-resolution, and synchronized. We demonstrate four ways the data could be utilized - (1) to discover and locate occluded objects using multiple sensors and modalities, (2) to associate audio events with their respective visual representations using both video and audio modes, (3) to track the amount of each type of object in a scene over time, and (4) to measure pedestrian speed using multiple synchronized camera views. In addition to these use cases, our data are available for other researchers to carry out analyses related to applying machine learning to understanding the urban environment (in which existing datasets may be inadequate), such as pedestrian-vehicle interaction modeling and pedestrian attribute recognition. Such analyses can help inform decisions made in the context of urban sensing and smart cities, including accessibility-aware urban design and Vision Zero initiatives.

Composing Digital Multimodal Notices Using the InShot Application as Learning Media

Digital technology has become very influential globally, particularly in developing learners' writing skills. This study investigated the implementation of the InShot application in creating digital multimodal notices. Moreover, it explored the strengths and weaknesses of the InShot app applied as the learning media in composing digital multimodal notices. The study used qualitative design (a case study) involving 27 students studying at a junior high school in West Java. Observations, open-ended questionnaires, and documents were applied to the data collection. The result shows that implementing the InShot app gives EFL learners optimal stimulus in composing digital multimodal notices and developing their writing skills. It is manifested in their composition of digital multimodal notices reflecting complete generic structure, correct linguistic features, and matching images expressing the full meanings of the notices. Besides, they show positive responses. In this case, they are excited and interested in composing digital multimodal notices using the Inshot app. They combined appropriate colours, fonts, and backgrounds to create the meanings of the notices. The Inshot app facilitates learners to integrate the meanings presented in verbal, visual, and audio modes harmoniously, so the purpose of the notices is complete. However, learners with less digital literacy need help integrating the appropriate modes in composing digital multimodal notices.

Action recognition based on element-level fine-grained multi-modal fusion

Traditional action recognition algorithms often only pay attention to video RGB features or optical flow features. These methods do not make good use of the audio information in the video. Based on RGB and optical flow characteristics, this paper introduces the processing of audio information, and classifies videos based on element-level fine-grained multi-modal fusion. Through experimental comparison, the accuracy of the multi-modal fusion algorithm proposed in this paper is improved by 7.38% on the HMDB51 dataset and 3.18% on the UCF101 dataset compared to the simple modal splicing. At the same time, it is proved that the introduction of audio modes can effectively improve the performance of the model.

Korean EFL Learner Preference for Text-Based Digital Composing During Emergency Remote Learning

This study utilized a longitudinal data collection to examine online factors of digital multimodal composing (DMC) preference and measure learner course satisfaction with digital composing modes in an online EFL communication course. The purpose of this research was to involve learners in a process of online, interactive, and multimodal curricular design during emergency remote learning due to the coronavirus pandemic. Innovative online technologies such as a new learning management system and digital educational components were implemented and used to quantitatively examine learners’ acceptance of technology. Korean learner preferences for textual and audio modes of DMC were indicated by Relative Advantage, Perceived Usefulness, and User Satisfaction factors. These factors also indicated an aversion to video-based DMC including video recordings and video responses as well as moderate concern for video conferencing. Qualitative findings revealed student concern for the constructs of Ease of Use and Using Video Modalities when transitioning to new online learning technologies.

Innovative Practices in Instructor E-feedback: A Case Study of E-feedback given in Three Linguistic Courses during the COVID 19 Pandemic

This study aimed to investigate the instructor’s electronic feedback practices during the COVID 19 Pandemic in terms of the nature of the content of e-feedback, the formulation, the challenges, and the multimodal nature of the instructor’s e-feedback. This study used a qualitative case study to obtain data from the instructor’s e-feedback in three linguistic courses as delivered, practiced by the single English language instructor. The instructor’s e-feedback via Blackboard and WhatsApp platform and the follow-up interview were analyzed qualitatively. The findings indicate that (1) the highest number of instructor’s e-feedback focused on global issues as compared to local issues, (2) the instructor composed his e-feedback in the form of eight main categories: explanations, suggestions, clarifications, questioning, repetitions, statements, praises, and commands, (3) the instructor used more screencasts for providing e-feedback, followed by written and audio modes respectively. The thematic analysis (4) revealed the instructor’s positive impression on providing e-feedback through these interactive modes (written, audio, and screencast) and a range of challenging issues such as students’ preference issues, technical issues, timing issues, financial and areal issues. This study is significant because it provides us with a comprehensive picture of the patterns of the feedback content, the formulation of the e-feedback, the multimodality of the instructor’s e-feedback, and the significant issues that emerged from the instructor’s e-feedback practices. However, further research should include a relative group of instructors to determine the impact of e-feedback on learners.

Specialist to non-specialist teleconsultations in chronic respiratory disease management: A systematic review.

BackgroundChronic respiratory diseases (CRD), are common public health problems with high prevalence, disability and mortality rates worldwide. Further uneven distribution of the health workforce is a major barrier to the effective diagnosis and treatment of CRDs. Teleconsultation between a specialist and non-specialist could possibly bridge the gap in access to health care and decrease CRD burden in remote areas. This review investigates the evidence for the effective use of specialist to non-specialist teleconsultation in the management of CRDs in remote areas and identifies instances of good practice and knowledge gaps.MethodsWe searched for articles till November 2020, which focused on specialist to non-specialist teleconsultations for CRD diagnosis or management. Two independent reviewers conducted the title and abstract screening and extracted data from the selected papers and the quality was assessed by Joanna Briggs Institute’s (JBI) tool. A descriptive and narrative approach was used due to the heterogeneous nature of the selected studies.ResultsWe found 1715, articles that met the initial search criteria, but after excluding duplicates and non-eligible articles, we included 10 research articles of moderate quality. These articles were from nine different studies, all of which, except one, were conducted in high-income countries. The studies reported results in terms of impact on the patients, and the health care providers including primary care physicians (PCP) and specialists. The teleconsulting systems used in all the selected papers primarily used audio modes in addition to other modes like the audio-video medium. The included studies reported primarily non-clinical outcomes including effectiveness, feasibility, acceptability and usability of the teleconsultation systems and only three described the clinical outcomes. The teleconsultation was predominantly conducted in the PCP’s office with the specialist located remotely.ConclusionsWe found relatively few, papers which explored specialist to non-specialist teleconsultation in management of CRDs, and no controlled trials. Two of the included papers described systems, which were used for other diseases in addition to the CRD. The available literature although not generalisable, encourages the use of specialist to non-specialist teleconsultation for diagnosis and management of CRDs.

English vowel perception by non-native speakers: impact of audio and visual training modalities

Perception of sounds of a second language (L2) presents difficulties for non-native speakers which can be improved with training (Bradlow, Pisoni, Akahane-Yamada & Tohkura, 1997; Logan, Lively & Pisoni, 1991; Iverson & Evans, 2009). The aim of this study was to compare three different English vowel perceptual training programmes using audio (A), audiovisual (AV) and video (V) modes in non-native speakers with Spanish as native language (L1). 47 learners of English with Spanish as L1 were allocated to three different vowel training groups (AT, AVT, VT) and were given five training sessions to assess their improvement in English vowel perception. Additionally, participants were recorded before and after training to measure their improvement in the production of English vowels. Results showed that participants improved their perception and production of English vowels regardless of their training modality with no evidence of a benefit of visual information. These results also suggest that there is a lot of individual differences in perception and production of L2 vowels which may be related to a complex relation between speech perceptual and production mechanisms.

Young Children’s Music Play Ideas: Two Case Studies of Syncretic Literacy Practice in Classroom and Home Settings

This critical enquiry into co-construction of meaning in music play uses applied literacy practices to explore children’s multimodal interactions. It shows evidence of cultural and social framing of their music making, their forms of organisation and ways of reinventing cultural knowledge during interaction. Using visual methodology and multimodal analysis, this study documents how children in diverse contexts intentionally transmit and redesign prior knowledge. Two case studies of diverse music activities, one in an early childhood rural setting and one in an inner-urban home setting, detail how two five-year-old children expanded communication with each other or with an adult using gestural, audio, spatial and visual modes as semiotic resources. These two multimodal experiences in music play are discussed to demonstrate how, in both situated events, young children demonstrated semiotic import of composing resources to transform prior knowledge in co-operative play. The activities illustrate how music play is a crucial element of everyday learning in early childhood settings. Teachers may promote learning by providing opportunities for children to co-construct and enact literacy in ways that transcend the curricular context. They expand literacy into larger worlds by recognising modes of gesture and spatial relations as students communicate life experiences through music play.

Boosting Innovation in an Italian Online University

This paper describes the process of boosting an innovative e-learning system in an online university in Italy. The system relies on a satellite-terrestrial telecommunication infrastructure and allows for different interaction types, including synchronous, asynchronous, textual, audio and video communication modes. The adoption of this infrastructure was preceded by a training initiative proposed to the university staff to favor its intake. The paper analyses the effects of both the training initiative and the technological innovation based on qualitative data derived from the observed differences between the pre-existing courses and their re-design and quantitative data tracked by the system during a pilot test that lasted eleven months. These data show a trend reversal in the e-learning approach, from a prevalence of transmissive mode to a more interactive one, although there is still a long way to go before more radical changes can take place.

Effects of Visual and Auditory Cues About Threat Location on Target Acquisition and Attention to Auditory Communications

This laboratory study examined the effects of visual, spatial language, and 3-D audio cues about target location on target acquisition performance and the recall of information contained in concurrent radio communications. Two baseline conditions were also included in the analysis: no cues (baseline 1) and target presence cues only (baseline 2). In modes in which target location cues were provided, 100% of the targets presented were acquired compared to 94% in baseline 1 and 95% in baseline 2. On average, targets were acquired 1.4 seconds faster in the visual, spatial language, and 3-D audio modes than in the baseline conditions, with times in the visual and 3-D audio modes being 1 second faster than those in spatial language. Overall workload scores were lower in the 3-D audio mode than in all other conditions except the visual mode. Less information (23%) was recalled from auditory communications in baseline 1 than in the other four conditions where attention could be directed to communications between target presentations.

Enhancing Online Education Using Collaboration Solutions

With the advances in Internet technologies, online education is fast gaining ground as an extension to traditional education. Webcast allows lectures conducted on campus to be viewed by students located at remote sites by streaming the audio and video content over Internet Protocol (IP) networks. However when used alone, webcast does not provide the form of user-to-user interaction that is so important in education. This article presents a collaboration system that can be used to facilitate interaction among users in a live webcast environment. The collaboration system supports real-time communication in both text and audio modes. To allow discussions to be carried out in an orderly manner, a virtual hand-raising scheme is implemented to put more control in the hands of the instructor. Using a topic browser, students can access content of previous discussions and use them as reference sources for any new question they may wish to raise. To allow offline viewing, the discussion content are automatically saved to the local computer at the end of each collaboration session.

Comparability of methods of observation for global ratings of heterosexual social skill and anxiety: A generalizability study

This research examined the degree to which skill and anxiety ratings vary as a function of audio, video, or live rating conditions. The question was addressed through a generalizability analysis which indicated that ratings of skill may generalize across audio, video, or live rating modes and across judges within a mode. However, results for anxiety show that while one can generalize across modes, there is a lack of comparability across judges within modes. This finding indicates the need for more judges in order to obtain acceptable levels of reliability for anxiety. Additional validity analyses suggest that the use of videotapes for ratings of skill maximizes the information obtained from role-play techniques for the assessment of heterosexual social performance.

Paired-Associate Learning Within and Across Sense Modalities and Involving Simultaneous and Sequential Presentations

Kale and Grosslight (1955) report a study in which the task was to write the correct Russian words when presented with their English equivalents. The presentation modes were the visual-print, the visualpictorial, the visual-pictorial with motion (the object representing the word was shown in motion), the audio and the kinesthetic (the S's vocalization of the learning materials), and their combinations. It was found that the presentation mode did make a difference in the amount of Russian vocabulary learned, with the visual-pictorial-with-motion mode being superior to the visual-print (p < .05) and to the complex combination of the visual-print, audio, kinesthetic, and visual-pictorial modes (p < .01). The audio mode was the least effective.