Multimodal Prototype Research Articles

Integrating multimodal contrastive learning with prototypical domain alignment for unsupervised domain adaptation of time series

Unsupervised domain adaptation (UDA) addresses the challenge of transferring knowledge from a labeled source domain to an unlabeled target domain. This task is particularly critical for time series data, characterized by unique temporal dynamics. However, existing methods often fail to capture these temporal dependencies, leading to domain discrepancies and loss of semantic information. In this study, we propose a novel framework for the unsupervised domain adaptation of time series (UDATS) that integrates Multimodal Contrastive Adaptation (MCA) and Prototypical Domain Alignment (PDA). MCA leverages image encoding techniques and prompt learning to capture complex temporal patterns while preserving semantic information. PDA constructs multimodal prototypes, combining visual and textual features to align target domain samples accurately. Our framework demonstrates superior performance across various application domains, including human activity recognition, mortality prediction, and fault detection. Experiments show our method effectively addresses domain discrepancies while preserving essential semantic content, outperforming state-of-the-art models.

Fine-Tuning for Few-Shot Image Classification by Multimodal Prototype Regularization

Fine-Tuning for Few-Shot Image Classification by Multimodal Prototype Regularization

MMPL-Net: multi-modal prototype learning for one-shot RGB-D segmentation

MMPL-Net: multi-modal prototype learning for one-shot RGB-D segmentation

Facile heart rate monitoring with multimodal capabilities

This paper presents the fabrication of a prototype with unique functionalities for monitoring human heart rate via simple insertion of fingertip on an infrared based sensor. The variation in the absorbance caused by the repeated pumping of oxygenated blood to various tissues within the body leads to distinct peaks and dips corresponding to the signal from the detector. Further, a MATLAB based Graphical User Interface (GUI) is developed to visualize this heart-beat signal as well as count the heart rate. In addition, the oxygen concentration can also be deciphered from the reported sensing scheme. The developed prototype seems to be in excellent match with standard heartrate monitoring device. It is believed that this multimodal prototype will pave way for more investigation towards development of sophisticated AI based analysis.

Multi-modal access method (eye-tracking + switch-scanning) for individuals with severe motor impairment: A preliminary investigation

ABSTRACT Individuals with severe motor impairments often require alternative means to access computers and communication technology. A range of alternative access devices exist; however, most rely on use of a single access modality. While this approach works for some individuals, it can be limiting for others. This study explored the use of a multi-modal prototype (eye-tracking + switch-scanning) on typing performance with a range of individuals with motor impairments. The multi-modal prototype was compared to eye-tracking alone for this study. Results indicated that the multi-modal prototype had significantly slower typing rate but significantly lower total errors compared to eye-tracking alone. Analysis of individual data revealed four subgroups of clinical relevance including individuals that 1) benefit from multi-modal, 2) benefit from eye-tracking, 3) demonstrate learning and 4) demonstrate fluctuating performance.

Improving Few-Shot Remote Sensing Scene Classification With Class Name Semantics

Few-shot remote sensing scene classification (FSRSSC) has been used for new class recognition in the presence of a limited number of labeled samples. The representation vector (prototype) of categories obtained using images only confronts some challenges, such as insufficient generalization when the number of samples is too small. To address this problem, we propose a new FSRSSC method based on prototype networks, named CNSPN, which combines semantic information of class names (name of the scene categories, such as aircraft, harbor, and bridge). First, CNSPN extracts semantics for class names using a pre-trained word-embedding model, which enriches the feature representation ability of the category at the source. Then, an enhanced fusion prototype is generated by fusing the semantic information of text and visual information in the image through a multimodal prototype fusion module (MPFM). Finally, the query image is classified by measuring the distance between the query sample and the visual prototype, and between the query sample and the fusion prototype. Comparative experiments on the NWPU-RESISC45 and RSD46-WHU datasets show that the proposed method significantly improves FSRSSC performance. Code is available at https://github.com/RS-CSU/CNSPN.git.

Operational Feasibility Analysis of the Multimodal Controller Working Position “TriControl”

Current Air Traffic Controller working positions (CWPs) are reaching their capacity owing to increasing levels of air traffic. The multimodal CWP prototype TriControl combines automatic speech recognition, multitouch gestures, and eye-tracking, aiming for more natural and improved human interaction with air traffic control systems. However, the prototype has not yet undergone systematic evaluation with respect to feasibility. This paper evaluates the operational feasibility, focusing on the system usability of the approach CWP TriControl and its fulfillment of operational requirements. Fourteen controllers took part in a simulation study to evaluate the TriControl concept. The active approach controllers among the group of participants served as the main core target subgroup. The ratings of all controllers in the TriControl assessment were, on average, generally in slight agreement, with just a few showing statistical significance. However, the active approach controllers performed better and rated the system much more positively. The active approach controllers were strongly positive regarding the system usability and acceptance of this early-stage prototype. Particularly, ease of use, user-friendliness, and learnability were perceived very positively. Overall, they were also satisfied with the command input procedure, and would use it for their daily work. Thus, the participating controllers encourage further enhancements to be made to TriControl.

Making the Invisible Visible—Strategies for Visualizing Underground Infrastructures in Immersive Environments

Visualization of underground infrastructure in an interactive 3D immersive environment is extremely important for efficient management of city’s infrastructure. This paper describes different geometric modelling approaches to illustrate appropriate visualization of such data. A multimodal prototype has been developed by exploiting different algorithms to render these invisible underground objects as part of an urban model. This prototype has been integrated in an immersive geographic information system (GIS), named MultiVis, for handheld iOS and Android devices. As a part of the study, three distinct strategies have been tested; the first is based on the use of transparencies to convey a sense of depth, the second relies on an image-space superposition of “ditches” on top of the rendered frame and the third is a world-space deformation of the elevation model that exposes the underground elements. Furthermore, a comparative user experience analysis of different techniques aimed to the geometrically accurate visualisation of utility networks and other underground facilities are performed and evaluated. It includes a set of user evaluations for different parameters of these techniques, which gives us an insight on how the proposed methods affect the experience and usability for technical and non-technical users.

Improving washing strategies of human mesenchymal stem cells using negative mode expanded bed chromatography

The use of human mesenchymal stem cells (hMSC) in clinical applications has been increasing over the last decade. However, to be applied in a clinical setting hMSC need to comply with specific requirements in terms of identity, potency and purity.This study reports the improvement of established tangential flow filtration (TFF)-based washing strategies, further increasing hMSC purity, using negative mode expanded bed adsorption (EBA) chromatography with a new multimodal prototype matrix based on core–shell bead technology.The matrix was characterized and a stable, expanded bed could be obtained using standard equipment adapted from what is used for conventional packed bed chromatography processes. The effect of different expansion rates on cell recovery yield and protein removal capacity was assessed. The best trade-off between cell recovery (89%) and protein clearance (67%) was achieved using an intermediate expansion bed rate (1.4).Furthermore, we also showed that EBA chromatography can be efficiently integrated on the already established process for the downstream processing (DSP) of hMSC, where it improved the washing efficiency more than 10-fold, recovering approximately 70% of cells after global processing. This strategy showed not to impact cell viability (>95%), neither hMSC's characteristics in terms of morphology, immunophenotype, proliferation, adhesion capacity and multipotent differentiation potential.

A wizard of oz component-based approach for rapidly prototyping and testing input multimodal interfaces

In this paper we present a novel approach for prototyping, testing and evaluating multimodal interfaces, OpenWizard. OpenWizard allows the designer and the developer to rapidly evaluate a non-fully functional multimodal prototype by replacing one modality or a composition of modalities that are not yet available by wizard of oz techniques. OpenWizard is based on a conceptual component-based approach for rapidly developing multimodal interfaces, an approach first implemented in the ICARE software tool and more recently in the OpenInterface tool. We present a set of wizard of oz components that are implemented in OpenInterface. While some wizard of oz (WoZ) components are generic to be reused for different multimodal applications, our approach allows the integration of tailored WoZ components. We illustrate OpenWizard using a multimodal map navigator.

Fusion of children's speech and 2D gestures when conversing with 3D characters

Most existing multi-modal prototypes enabling users to combine 2D gestures and speech input are task-oriented. They help adult users solve particular information tasks often in 2D standard Graphical User Interfaces. This paper describes the NICE Andersen system, which aims at demonstrating multi-modal conversation between humans and embodied historical and literary characters. The target users are 10–18 years old children and teenagers. We discuss issues in 2D gesture recognition and interpretation as well as temporal and semantic dimensions of input fusion, ranging from systems and component design through technical evaluation and user evaluation with two different user groups. We observed that recognition and understanding of spoken deictics were quite robust and that spoken deictics were always used in multi-modal input. We identified the causes of the most frequent failures of input fusion and suggest possible improvements for removing these errors. The concluding discussion summarises the knowledge provided by the NICE Andersen system on how children gesture and combine their 2D gestures with speech when conversing with a 3D character, and looks at some of the challenges facing theoretical solutions aimed at supporting unconstrained speech/2D gesture fusion.