Multimodal Information Presentation Research Articles

The factors involved in surgical decision-making in younger women diagnosed with breast cancer in Aotearoa New Zealand: A qualitative analysis.

Women diagnosed with breast cancer must make important surgical decisions. The decision-making process for younger women is complex, with this group more likely to have an advanced diagnosis and life-stage considerations that can impact on treatment. This study investigated the decision-making process of women aged <50 years who had undergone breast cancer surgery within the preceding 12 months in Aotearoa New Zealand. Twelve women participated in semi-structured qualitative interviews to explore the factors that influenced treatment decisions. Thematic analysis resulted in three themes. Fear was the main concept identified as the primary influence on initial decision-making. Good quality shared decision-making between patient and clinician was found to provide essential support during the diagnosis to treatment period. In addition, women expressed a need for multi-modal presentation of medical information and more material reflecting younger women. These findings inform provision for younger women making surgical decisions when diagnosed with breast cancer.

Development and Evaluation of Refreshable Braille Display and Active Touch-Reading System for Digital Reading of the Visually Impaired.

The traditional way of reading through Braille books is constraining the reading experience of blind or visually impaired (BVI) in the digital age. In order to improve the reading convenience of BVI, this paper proposes a low-cost and refreshable Braille display device, and solves the problems of high energy consumption and low latching force existing in existing devices. Further, the Braille display device was combined with the 3D Systems Touch device to develop an active Braille touch-reading system for digital reading of BVI with the help of the CHAI3D virtual environment. Firstly, according to the actual needs of BVI to touch and read the Braille dots, this paper utilizes the beam structure to provide a full latching function for the raised Braille dot without energy consumption. Through theoretical derivation and finite element analysis, the performance of the Braille dot actuator is optimized to provide sufficient feedback force and latching force for finger's touch-reading. Then, this paper designs a virtual Braille interactive environment based on the CHAI3D, and combines the sense of touch with audio to effectively improve the recognition accuracy and reading efficiency of BVI for Braille through the multi-modal presentation of Braille information. The performance test results of the device show that the average lifting force of the Braille dot actuator is 101.67 mN, the latching force is over 5 N, and the average refresh frequency is 17.1 Hz, which meets the touch-reading needs of BVI. User experiments show that the average accuracy rate of BVI subjects in identifying digitized Braille is 95.5%, and subjects have a high subjective evaluation of the system.

Key factors in the decision-making process for mastectomy alone or breast reconstruction: A qualitative analysis

Breast cancer is the most common cancer to affect New Zealand women. Women diagnosed face several decisions regarding surgical treatment, including whether to undergo lumpectomy, mastectomy, or breast reconstruction. Reconstructive surgery adds an additional layer of complexity, with several reconstructive options, each associated with differing surgical and recovery times. Furthermore, surgical decisions are often made under time-pressure and significant diagnostic distress, therefore provision of good information to support decision-making is crucial to adequately inform women of their options. We interviewed 24 women who had undergone breast surgery within the preceding 12 months to assess the key factors leading to their decision to opt for their chosen surgical procedure. Interviews revealed that decision-making was complex and involved multiple factors. Women were ultimately confronted with assessing feminine identity versus survival. Whether opting for breast reconstruction or not, women were fearful of what surgery would involve and how their reconstructed breast or mastectomy scar might look following surgery. Shared decision-making between patient and clinician can mitigate this fear and provide women with a sense of autonomy over their health decisions. Provision of visual depictions of surgical outcomes was not routinely provided to those interviewed but was expressed as important to help women manage surgical expectations. Therefore our findings support the multi-modal presentation of diagnostic and treatment information to support decision-making. Likewise, women reported feeling unsupported in their decision not to undergo breast reconstruction, suggesting a need to develop resources to provide women with positive discussions about ‘going flat’.

Promotion of driver compliance with V2V information in car-following tasks via multimodal display

The success of vehicle-to-vehicle (V2V) communication in improving the driving safety and efficiency primarily depends on driver compliance with the transmitted information. In this study, driver compliance with V2V information in simulated car-following tasks was measured and analyzed under different information display settings, such as baseline without information, visual information only, and multimodal information. Car-following data were collected from 30 participants in a driving simulation environment. The results revealed that V2V information could significantly reduce the driver's response time to the lead car’s speed change, resulting in an increase in the minimum time-to-collision when the lead car slowed down. Driver compliance was significantly higher under multimodal information settings than when using visual information alone. Moreover, driver compliance was higher when facing a decelerating lead car than when facing an accelerating lead car. No significant differences in driver compliance between the deceleration levels was observed; however, differences existed among different car-following preference groups. The findings of this study can be applied to V2V car-following models to improve the simulation accuracy for connected-vehicle traffic. Additionally, multimodal information presentation can provide a reference for V2V communication design.

The Impacts of the use of Thematic &amp; Chronologic Multi-modal Information Representation on Sequential and Global Students’ Historical Understanding

This study examined the two different modes of multi-modes information presentations that affected sequential and global learners' history understanding: thematic and chronological. A total of 134 secondary schools’ students were enrolled (69 learning in chronological mode, 65 learning in thematic mode). Before the start of the treatment session, students were given a pre-test. The results showed that multimodal information presentation did not have a significantly great impact on historical learning or between pupils who learn in chronological and thematic ways. The chronological frame of reference technique, which reflected an interactive timeline, was reported to have supported students' sequential learning in chronological mode. Students who learned in thematic mode, on the other hand, had greater gains in points of historical understanding than students who learned in chronological mode. Students who exhibited a significant difference in their assessment of historical understanding proved the effectiveness of multimedia information presentation in acquiring history-related abstract ideas.

Integrating Cognitive Dysfunction Accommodation Strategies Into Behavioral Interventions for Persons on Medication for Opioid Use Disorder.

BackgroundCognitive dysfunction is disproportionately prevalent among persons with opioid use disorder (OUD). Specific domains of cognitive dysfunction (attention, executive functioning, memory, and information processing) may significantly impede treatment outcomes among patients on medication for OUD (MOUD). This limits patient's ability to learn, retain, and apply information conveyed in behavioral intervention sessions. Evidence-based accommodation strategies have been integrated into behavioral interventions for other patient populations with similar cognitive profiles as persons with OUD; however, the feasibility and efficacy of these strategies have not yet been tested among patients on MOUD in a drug treatment setting.MethodsWe conducted a series of focus groups with 25 key informants (10 drug treatment providers and 15 patients on MOUD) in a drug treatment program in New Haven, CT. Using an inductive approach, we examined how cognitive dysfunction impedes participant's ability to retain, recall, and utilize HIV prevention information in the context of drug treatment.ResultsTwo main themes capture the overall responses of the key informants: (1) cognitive dysfunction issues and (2) accommodation strategy suggestions. Subthemes of accommodation strategies involved suggestions about particular evidence-based strategies that should be integrated into behavioral interventions for persons on MOUD. Specific accommodation strategies included: use of a written agenda, mindfulness meditation, multi-modal presentation of information, hands-on demonstrations, and a formal closure/summary of sessions.ConclusionsAccommodation strategies to compensate for cognitive dysfunction were endorsed by both treatment providers and patients on MOUD. These accommodation strategies have the potential to enhance the efficacy of behavioral interventions to reduce HIV transmission among persons on MOUD as well as addiction severity, and overdose.

Development and application of a multi-modal task analysis to support intelligent tutoring of complex skills

BackgroundContemporary work in the design and development of intelligent training systems employs task analysis (TA) methods for gathering knowledge that is subsequently encoded into task models. These task models form the basis of intelligent interpretation of student performance within education and training systems. Also referred to as expert models, they represent the optimal way(s) of performing a training task. Within Intelligent Tutoring Systems (ITSs), real-time comparison of trainee task performance against the task model drives automated assessment and interactive support (such as immediate feedback) functionality. However, previous task analysis (TA) methods, including various forms of cognitive task analysis (CTA), may not be sufficient to support identification of the detailed design specifications required for the development of an ITS for a complex training task incorporating multiple underlying skill components, as well as multi-modal information presentation, assessment, and feedback modalities. Our current work seeks to develop an ITS for training Robotic Assisted Laparoscopic Surgery (RALS), a complex task domain that requires a coordinated utilization of integrated cognitive, psychomotor, and perceptual skills.ResultsIn this paper, we describe a methodological extension to CTA, referred to as multi-modal task analysis (MMTA) that elicits and captures the nuances of integrated and isolated cognitive, psychomotor, and perceptual skill modalities as they apply to training and performing complex operational tasks. In the current case, we illustrate the application of the MMTA method described here to RALS training tasks. The products of the analysis are quantitatively summarized, and observations from a preliminary qualitative validation are reported.ConclusionsWe find that iterative use of the described MMTA method leads to sufficiently complete and robust task models to support encoding of cognitive, psychomotor, and perceptual skills requisite to training and performance of complex skills within ITS task models.

What You Don't Notice Can Harm You: Age-Related Differences in Detecting Concurrent Visual, Auditory, and Tactile Cues.

Objective This research sought to determine whether people can perceive and process three nonredundant (and unrelated) signals in vision, hearing, and touch at the same time and how aging and concurrent task demands affect this ability. Background Multimodal displays have been shown to improve multitasking and attention management; however, their potential limitations are not well understood. The majority of studies on multimodal information presentation have focused on the processing of only two concurrent and, most often, redundant cues by younger participants. Method Two experiments were conducted in which younger and older adults detected and responded to a series of singles, pairs, and triplets of visual, auditory, and tactile cues in the absence (Experiment 1) and presence (Experiment 2) of an ongoing simulated driving task. Detection rates, response times, and driving task performance were measured. Results Compared to younger participants, older adults showed longer response times and higher error rates in response to cues/cue combinations. Older participants often missed the tactile cue when three cues were combined. They sometimes falsely reported the presence of a visual cue when presented with a pair of auditory and tactile signals. Driving performance suffered most in the presence of cue triplets. Conclusion People are more likely to miss information if more than two concurrent nonredundant signals are presented to different sensory channels. Application The findings from this work help inform the design of multimodal displays and ensure their usefulness across different age groups and in various application domains.

Positioning Learning Support System by Multimodal Information Presentation for Violin Beginners

Positioning Learning Support System by Multimodal Information Presentation for Violin Beginners

Multimodal Information Presentation in Support of NextGen Operations

ABSTRACTObjective: This study examined the effectiveness of visual, auditory, tactile, and redundant auditory-visual information presentation in the context of a medium-fidelity ‘Next Generation Air Transportation System’ (NextGen) flight simulation.Background: Data overload, especially in the visual channel, and associated breakdowns in monitoring represent a major challenge in aviation. These problems are expected to worsen with NextGen, which will require pilots to manage increased amounts of data and adopt new responsibilities. The introduction of multimodal interfaces (interfaces that distribute information across multiple sensory channels) has been proposed as a means to offload the overburdened visual channel and thus address data overload.Method: Experienced commercial airline pilots completed 2 scenarios using a medium-fidelity flight simulator. For each scenario, NextGen tasks and events were presented either using technology that is currently available (visual and auditory displays) or technology proposed as part of NextGen design concepts (i.e., tactile and redundant displays). Performance was measured based on response time and accuracy.Results: Faster responses were observed for redundant displays, compared to either vision or audition alone. No significant benefit of redundancy was found for accuracy and workload did not mediate redundancy effects. For traffic events, there were faster response times with tactile displays, but higher response accuracy with auditory displays.Conclusion: The findings from this research add to the knowledge base in multimodal information processing and can inform the design of displays for NextGen operations.

Auditory-Visual Redundancy in Vehicle Control Interruptions: Two Meta-analyses

Two novel versions of a meta analysis were employed to assess the conditions of ongoing vehicle control task simulations in which (1) auditory presentation of an interrupting task were beneficial over visual presentations and (2) redundant (av) presentation was better than single modality presentation (providing redundancy gain). Altogether 29 studies were identified. The results revealed that the interrupting task benefited from auditory presentation, but the ongoing task (visual vehicle control task) generally did not. Performance of the visual interrupting task was slightly hindered by separation from the ongoing task. The redundancy analysis revealed that the interrupting task benefited from redundancy when it involved spatial localization and alerting and the accuracy of verbal communications; but suffered when speed of the verbal communications response was measured, and when the two visual channels were separated. Implications for multi-modal presentation of information on vehicle workstations are discussed. Language: en

Crossmodal Links in Attention in the Driving Environment: The Roles of Cueing Modality, Signal Timing, and Workload

Multimodal information presentation has been proposed as a means to support timesharing in complex data-rich environments. To ensure the effectiveness of this approach, it is necessary to consider performance effects of recently discovered crossmodal spatial and temporal links in attention, as well as their interaction with other performance-shaping factors. The main goals of this research were to confirm that performance effects of crossmodal links in spatial attention scale to complex environments and to examine how these effects vary as a function of cue modality, signal timing, and workload. In the present study, set in a driving simulation, spatially valid and invalid auditory and tactile cues preceded the presentation of visual targets at various stimulus-onset asynchronies and under different levels of workload induced by simulated wind gusts of varied intensity. The findings from this experiment confirm that visual target identification accuracies and response times are, overall, more accurate and faster when validly-cued. Significant interactions were found between cue validity, stimulus onset asynchrony (SOA), and cue modality, such that valid tactile cueing is most beneficial at shorter (100–200 ms) SOAs, while valid auditory cueing resulted in faster responses than invalid cueing at 500 ms SOAs, but slower responses at 1000 ms SOAs. Tactile error rates were significantly higher than auditory error rates at various interactions of modality and SOA. These findings were robust across all workload conditions. They highlight the need for context-sensitive information presentation and can inform the design of multimodal interfaces for a wide range of application domains.

Creation and evaluation of a multi-sensory virtual assembly environment

The multi-modal information presentation, integrated into the virtual environment (VE), has potential for stimulating different senses, improving the user’s impression of immersion, and increasing the amount of information that is accepted and processed by the user’s perception system. The increase of the useful feedback information may reduce the user’s cognitive load, thus enhancing the user’s efficiency and performance while interacting with VEs. This paper presents our creation of a multi-sensory virtual assembly environment (VAE) and the evaluation of the effects of multi-sensory feedback on the usability. The VAE brings together complex technologies such as constraint-based assembly simulation, optical motion tracking technology, and real-time 3D sound generation technology around a virtual reality workbench and a common software platform. The usability evaluation is in terms of its three attributes: efficiency of use, user satisfaction, and reliability. These are addressed by using task completion times (TCTs), questionnaires, and human performance error rates (HPERs), respectively. Two assembly tasks have been used to perform the experiments, using sixteen participants. The outcomes showed that the multi-sensory feedback could improve the usability. They also indicated that the integrated feedback offered better usability than either feedback used in isolation. Most participants preferred the integrated feedback to either feedback (visual or auditory) or no feedback. The participants’ comments demonstrated that nonrealistic or inappropriate feedback had negative effects on the usability, and easily made them feel frustrated. The possible reasons behind the outcomes are also analysed by using a unifying human computer interaction framework. The implications, concluded from the outcomes of this work, can serve as useful guidelines for improving VE system design and implementation.

A framework for the intelligent multimodal presentation of information

Intelligent multimodal presentation of information aims at using several communication modalities to produce the most relevant user outputs. This problem involves different concepts related to information structure, interaction components (modes, modalities, devices) and context. In this paper we study the influence of interaction context on system outputs. More precisely, we propose a conceptual model for intelligent multimodal presentation of information. This model, called WWHT, is based on four concepts: “What”, “Which”, “How” and “Then”. These concepts describe the life cycle of a multimodal presentation from its “birth” to its “death” including its evolution. On the basis of this model, we present the ELOQUENCE software platform for the specification, the simulation and the execution of output multimodal systems. Finally, we describe two applications of this framework: the first one concerns the simulation of an incoming call in an intelligent mobile phone and the second one is related to a task of marking out a target on the ground in a fighter plane cockpit.

Multimodal information presentation: Design guidance and research challenges

Multimodal interfaces have gained considerable interest in recent years. This trend can be expected to continue as designers try to address challenges like data overload, the need for improved performance of recognition-based systems, a greater sense of immersion in virtual-reality environments, and support for time sharing and attention management in a variety of complex real-world domains. To ensure the robustness and effectiveness of multimodal interfaces, a number of design guidelines have been proposed. The present paper will review guidelines that focus on multimodal information presentation (as opposed to multimodal system input), and it will discuss important research needs in this area, including the exploration of possible uses and benefits of rarely used sensory channels (such as touch and olfaction), the consideration of crossmodal links in attention, the top–down modulation of multimodal information processing, and the need for adaptive and adaptable multimodal interfaces. Relevance to industry Multimodal interfaces will likely find their way into an increasing number of complex data-rich environments. It is critical for engineers and designers to be aware of benefits, requirements, and constraints associated with the design of multimodal displays to ensure their effectiveness and avoid pitfalls of this approach to information presentation.

Towards a multilingual, multimedia and multimodal digital library platform

The China-US Million Book Digital Library Project (Million Book Project) is an international cooperation program between China and the US. However, one million digitized books are considered not to be the ultimate goal of the project, but a first step towards universal access to human knowledge. In particular, there are four challenges about the new way to analyze, process, operate, visualize and interact with digital media resource in this library. To tackle these challenges, North China Centre of Million Book Project (in Chinese Academy of Sciences) has initiated several innovative research projects in areas such as multimedia content analysis and retrieval, bilingual services, multimodal information presentation, and knowledge-based or- ganization and services. In this keynote speech, we simply review our work in these areas, and argue that by technological co- operation with these innovation research topics, the project will develop a top-level digital library platform for the million book library.

Graded and Multimodal Interruption Cueing in Support of Preattentive Reference and Attention Management

A decade ago, attention management was considered by some “the least explored frontier in cognitive science and human-machine cooperation” (Woods et al., 1994). Today, at least one aspect of attention management — interruption handling — still poses a major challenge for the design of human-machine systems and computer-supported collaborative work. Successful interruption management requires that both unintentional dismissals and preemptive integrations of interruption signals are avoided (Latorella, 1999). One means of achieving this goal is to support preattentive reference, i.e., the processing of interruption signals that occurs before attentional selection. Operators need to be provided with at least partial information about the nature and cognitive requirements of a potential interruption, and this information should be presented in a way that allows for peripheral access. This paper will discuss and illustrate how currently underutilized graded and multimodal information presentation could help accomplish this goal and support various stages of the overall interruption management process.

Supporting Timesharing and Interruption Management Through Multimodal Information Presentation

Operators in complex event-driven domains often need to perform multiple concurrent tasks and handle competing attentional demands, such as interruptions by other human or machine agents. This study examined the effectiveness of distributing tasks across various sensory channels and presenting information on the nature of an interruption task to support timesharing and attention management. Participants performed a visually demanding simulated Air Traffic Control (ATC) task involving Data Link communication. At times, an interruption task was introduced, which consisted of counting subsets of signals that were presented in visual, auditory, or tactile form. Half of the subjects automatically received information on the modality and urgency of these pending interruption tasks whereas the other participants had the option to request this information. Within-subject variables in this study included ATC-related workload and the frequency and priority of interruption tasks. High-priority tasks had to be performed immediately whereas low-priority tasks could be delayed for up to two minutes. The results show that information about the nature of pending tasks supported participants in scheduling and timesharing more effectively. They were able to avoid intramodal interference and scanning costs associated with performing the ATC task concurrently with a visual interruption task. Crossmodal interference was lowest for auditory interruption tasks. Overall, these findings illustrate the benefits of multimodal information presentation and more informative interruption signals.

Prosody modelling in concept-to-speech generation: methodological issues

Restricted accessMoreSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InRedditEmail Cite this article McKeown Kathleen R. and Pan Shimei 2000Prosody modelling in concept-to-speech generation: methodological issuesPhil. Trans. R. Soc. A.3581419–1431http://doi.org/10.1098/rsta.2000.0595SectionRestricted accessProsody modelling in concept-to-speech generation: methodological issues Kathleen R. McKeown Kathleen R. McKeown Department of Computer Science, Columbia University, New York, NY 10027, USA Google Scholar Find this author on PubMed Search for more papers by this author and Shimei Pan Shimei Pan Department of Computer Science, Columbia University, New York, NY 10027, USA Google Scholar Find this author on PubMed Search for more papers by this author Kathleen R. McKeown Kathleen R. McKeown Department of Computer Science, Columbia University, New York, NY 10027, USA Google Scholar Find this author on PubMed Search for more papers by this author and Shimei Pan Shimei Pan Department of Computer Science, Columbia University, New York, NY 10027, USA Google Scholar Find this author on PubMed Search for more papers by this author Published:15 April 2000https://doi.org/10.1098/rsta.2000.0595AbstractWe explore three issues for the development of concept–to–speech (CTS) systems. We identify information available in a language–generation system that has the potential to impact prosody; investigate the role played by different corpora in CTS prosody modelling; and explore different methodologies for learning how linguistic features impact prosody. Our major focus is on the comparison of two machine learning methodologies: generalized rule induction and memory–based learning. We describe this work in the context of multimedia abstract generation of intensive care (MAGIC) data, a system that produces multimedia briefings of the status of patients who have just undergone a bypass operation. Previous Article VIEW FULL TEXT DOWNLOAD PDF FiguresRelatedReferencesDetailsCited by Jordan D and Rose S (2009) Multimedia Abstract Generation of Intensive Care Data: The Automation of Clinical Processes Through AI Methodologies, World Journal of Surgery, 10.1007/s00268-009-0319-5, 34:4, (637-645), Online publication date: 1-Apr-2010. Theune M, Heylen D and Nijholt A (2005) Generating Embodied Information Presentations Multimodal Intelligent Information Presentation, 10.1007/1-4020-3051-7_3, (47-67), . Xydas G, Spiliotopoulos D and Kouroupetroglou G (2004) Modeling Prosodic Structures in Linguistically Enriched Environments Text, Speech and Dialogue, 10.1007/978-3-540-30120-2_66, (521-528), . Bulyko I and Ostendorf M (2002) Efficient integrated response generation from multiple targets using weighted finite state transducers, Computer Speech & Language, 10.1016/S0885-2308(02)00023-2, 16:3-4, (533-550), Online publication date: 1-Jul-2002. Jordan D, McKeown K, Concepcion K, Feiner S and Hatzivassiloglou V (2001) Generation and Evaluation of Intraoperative Inferences for Automated Health Care Briefings on Patient Status After Bypass Surgery, Journal of the American Medical Informatics Association, 10.1136/jamia.2001.0080267, 8:3, (267-280), Online publication date: 1-May-2001. This Issue15 April 2000Volume 358Issue 1769Discussion Meeting Issue ‘Computers, language and speech: formal theories and statistical data’ organized by the Royal Society and the British Academy Article InformationDOI:https://doi.org/10.1098/rsta.2000.0595Published by:Royal SocietyPrint ISSN:1364-503XOnline ISSN:1471-2962History: Published online15/04/2000Published in print15/04/2000 License: Citations and impact KeywordsConcept–To–Speech Generation Speech Synthesis Natural Language Generation Machine Learning

Multi-Modal Cockpit Warnings: Pictures. Words. or Both?

This paper examines the requirement for, and benefits of, multi-modal information presentation in cockpit warning systems. It also attempts to provide an account of the nature and levels of processing involved when information from one source is integrated with a supposedly redundant additional source to provide a performance gain in choice reaction time (RT) tasks, and its applicability to other cockpit systems. An experiment is described which used warning/caution ‘icons’ (pictorial representations of danger situations) and verbal warning messages, both singly and in combination. The visual icons were generated by RAF aircrew, using an iterative design process, as being meaningful pictorial representations of real-world warnings. Subjects were required to identify whether the situations presented warnings i.e. high priority/immediate action or cautions i.e. low priority/immediate awareness. The results obtained showed a significant decrease in response latencies when correlated bi-modal information was given as compared to the uni-modal conditions. The high level of abstraction of these icons strongly implied that the performance gains occurring must be as the result of the integration of ‘information’ rather than ‘data’. Subjective Situational Awareness Rating Technique (SART) scores also showed that benefits may also be accrued through reduced workload and increased depth of understanding. The results of these experiments are considered in terms of current Information Processing and Neural Network theories and an attempt to provide a cognitive model of this integrality effect is also described.