Multimodal Input Research Articles

Sleep Stage Classification With Multi-Modal Fusion and Denoising Diffusion Model.

Sleep stage classification plays a crucial role in sleep quality assessment and sleep disorder prevention. Nowadays, many studies have developed algorithms for this purpose, but they still face two challenges. The first is noise in physiological signals from various devices. The second challenge is that most studies simply concatenate multi-modal features without considering their correlations. To this end, we propose a framework, namely Diff-SleepNet, to efficiently classify sleep stages from multi-modal input. This framework begins with a diffusion model with peak signal-to-noise ratio (PNSR) loss function that adaptively filters noise. The filtered signals are then transformed into a multi-view spectrum through data pre-processing. These spectra are processed by a transformer-based backbone to extract multi-modal features. The production is fed into the following multi-scale attention module for robust feature fusion. The sleep stage category is finally determined by a fully connected layer. Our framework is trained and validated on three typical datasets, i.e., SHHS, Sleep-EDF-SC, and Sleep-EDF-X. Experimental results demonstrate that it is effective and has advantages over other peer methods.

Decouple Ego-View Motions for Predicting Pedestrian Trajectory and Intention.

Pedestrian trajectory prediction is a critical component of autonomous driving in urban environments, allowing vehicles to anticipate pedestrian movements and facilitate safer interactions. While egocentric-view-based algorithms can reduce the sensing and computation burdens of 3D scene reconstruction, accurately predicting pedestrian trajectories and interpreting their intentions from this perspective requires a better understanding of the coupled vehicle (camera) and pedestrian motions, which has not been adequately addressed by existing models. In this paper, we present a novel egocentric pedestrian trajectory prediction approach that uses a two-tower structure and multi-modal inputs. One tower, the vehicle module, receives only the initial pedestrian position and ego-vehicle actions and speed, while the other, the pedestrian module, receives additional prior pedestrian trajectory and visual features. Our proposed action-aware loss function allows the two-tower model to decompose pedestrian trajectory predictions into two parts, caused by ego-vehicle movement and pedestrian movement, respectively, even when only trained on combined ego-view motions. This decomposition increases model flexibility and provides a better estimation of pedestrian actions and intentions, enhancing overall performance. Experiments on three publicly available benchmark datasets show that our proposed model outperforms all existing algorithms in ego-view pedestrian trajectory prediction accuracy.

A Unified Multi-Modality Fusion Framework for Deep Spatio-Temporal-Spectral Feature Learning in Resting-State fMRI Denoising.

Resting-state functional magnetic resonance imaging (rs-fMRI) is a commonly used functional neuroimaging technique to investigate the functional brain networks. However, rs-fMRI data are often contaminated with noise and artifacts that adversely affect the results of rs-fMRI studies. Several machine/deep learning methods have achieved impressive performance to automatically regress the noise-related components decomposed from rs-fMRI data, which are expressed as the pairs of a spatial map and its associated time series. However, most of the previous methods individually analyze each modality of the noise-related components and simply aggregate the decision-level information (or knowledge) extracted from each modality to make a final decision. Moreover, these approaches consider only the limited modalities making it difficult to explore class-discriminative spectral information of noise-related components. To overcome these limitations, we propose a unified deep attentive spatio-spectral-temporal feature fusion framework. We first adopt a learnable wavelet transform module at the input-level of the framework to elaborately explore the spectral information in subsequent processes. We then construct a feature-level multi-modality fusion module to efficiently exchange the information from multi-modality inputs in the feature space. Finally, we design confidence-based voting strategies for decision-level fusion at the end of the framework to make a robust final decision. In our experiments, the proposed method achieved remarkable performance for noise-related component detection on various rs-fMRI datasets.

Sensor Independent Cloud and Shadow Masking With Partial Labels and Multimodal Inputs

Sensor Independent Cloud and Shadow Masking With Partial Labels and Multimodal Inputs

Large Language Models: A Comprehensive Guide for Radiologists

Large language models (LLMs) have revolutionized the global landscape of technology beyond the field of natural language processing. Owing to their extensive pre-training using vast datasets, contemporary LLMs can handle tasks ranging from general functionalities to domain-specific areas, such as radiology, without the need for additional fine-tuning. Importantly, LLMs are on a trajectory of rapid evolution, addressing challenges such as hallucination, bias in training data, high training costs, performance drift, and privacy issues, along with the inclusion of multimodal inputs. The concept of small, on-premise open source LLMs has garnered growing interest, as fine-tuning to medical domain knowledge, addressing efficiency and privacy issues, and managing performance drift can be effectively and simultaneously achieved. This review provides conceptual knowledge, actionable guidance, and an overview of the current technological landscape and future directions in LLMs for radiologists.

Autoencoder-Based Collaborative Attention GAN for Multi-Modal Image Synthesis

Multi-modal images are required in a wide range of practical scenarios, from clinical diagnosis to public security. However, certain modalities may be incomplete or unavailable because of the restricted imaging conditions, which commonly leads to decision bias in many real-world applications. Despite the significant advancement of existing image synthesis techniques, learning complementary information from multi-modal inputs remains challenging. To address this problem, we propose an autoencoder-based collaborative attention generative adversarial network (ACA-GAN) that uses available multi-modal images to generate the missing ones. The collaborative attention mechanism deploys a single-modal attention module and a multi-modal attention module to effectively extract complementary information from multiple available modalities. Considering the significant modal gap, we further developed an autoencoder network to extract the self-representation of target modality, guiding the generative model to fuse target-specific information from multiple modalities. This considerably improves cross-modal consistency with the desired modality, thereby greatly enhancing the image synthesis performance. Quantitative and qualitative comparisons for various multi-modal image synthesis tasks highlight the superiority of our approach over several prior methods by demonstrating more precise and realistic results. The source code will be available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/caohaifang123/ACA-GAN</uri> .

A Multimodal Assistive-Robotic-Arm Control System to Increase Independence After Tetraplegia.

Following tetraplegia, independence for completing essential daily tasks, such as opening doors and eating, significantly declines. Assistive robotic manipulators (ARMs) could restore independence, but typically input devices for these manipulators require functional use of the hands. We created and validated a hands-free multimodal input system for controlling an ARM in virtual reality using combinations of a gyroscope, eye-tracking, and heterologous surface electromyography (sEMG). These input modalities are mapped to ARM functions based on the user's preferences and to maximize the utility of their residual volitional capabilities following tetraplegia. The two participants in this study with tetraplegia preferred to use the control mapping with sEMG button functions and disliked winking commands. Non-disabled participants were more varied in their preferences and performance, further suggesting that customizability is an advantageous component of the control system. Replacing buttons from a traditional handheld controller with sEMG did not substantively reduce performance. The system provided adequate control to all participants to complete functional tasks in virtual reality such as opening door handles, turning stove dials, eating, and drinking, all of which enable independence and improved quality of life for these individuals.

A Two-Stage Stacked Transformer Framework for Multimodal Sentiment Analysis

Transformer-based methods have achieved superior performance in multimodal sentiment analysis (MSA). However, recent studies have used it more to focus on the potential mutual adaptation between unimodal modalities, while ignoring intra-modal interactions and complementarity between potential fusion representations. To address this problem, this study proposes a two-stage stacked transformer framework for MSA. The framework decomposes the fusion into two stages, each of which concentrates on a subset of multimodal signals to simultaneously capture the communication information between the unimodal modalities and the interaction information between fusion representations. Further, stacked transformers are the core components of the framework. Two transformer layers are used to model the cross-modal interaction and intra-modal interaction of multimodal input. For cross-modal attention, we propose an attention weight accumulation mechanism to further improve the ability of the framework. Experimental results on 3 benchmark datasets (MOSI [Multimodal Opinion-level Sentiment Intensity], MOSEI [Multimodal Opinion Sentiment and Emotion Intensity], and SIMS) show performance superior or comparable to the state-of-the-art models, thus demonstrating the effectiveness of the proposed framework.

An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy Tasks.

Large Language Models (LLMs) like GPT-4 which support multimodal input (i.e., prompts containing images in addition to text) have immense potential to advance visualization research. However, many questions exist about the visual capabilities of such models, including how well they can read and interpret visually represented data. In our work, we address this question by evaluating the GPT-4 multimodal LLM using a suite of task sets meant to assess the model's visualization literacy. The task sets are based on existing work in the visualization community addressing both automated chart question answering and human visualization literacy across multiple settings. Our assessment finds that GPT-4 can perform tasks such as recognizing trends and extreme values, and also demonstrates some understanding of visualization design best-practices. By contrast, GPT-4 struggles with simple value retrieval when not provided with the original dataset, lacks the ability to reliably distinguish between colors in charts, and occasionally suffers from hallucination and inconsistency. We conclude by reflecting on the model's strengths and weaknesses as well as the potential utility of models like GPT-4 for future visualization research. We also release all code, stimuli, and results for the task sets at the following link: https://doi.org/10.17605/OSF.IO/F39J6.

Fusion-Embedding Siamese Network for Light Field Salient Object Detection

Light field salient object detection (SOD) has shown remarkable success and gained considerable attention from the computer vision community. Existing methods usually employ a single-/two-stream network to detect saliency. However, these methods can only handle up to two different modalities at a time, preventing them from being able to fully explore the rich information in multi-modal light field derived data. To address this, we propose the first joint multi-modal learning framework, called FES-Net, for light field SOD, which can take rich inputs not limited to two modalities. Specifically, we propose an attention-aware adaptation module to first transform the multi-modal inputs for use in our joint learning framework. The transformed inputs are then fed to a Siamese network along with multiple embedded feature fusion modules to extract informative multi-modal features. Finally, we predict saliency maps from the high-level extracted features using a saliency decoder module. Our joint multi-modal learning framework effectively resolves the limitations of existing methods, providing efficient and effective multi-modal learning that can fully explore the valuable information in light field data for accurate saliency detection. Furthermore, we improve the performance by introducing the Transformer as our backbone network. To the best of our knowledge, the improved version of our model, called FES-Trans, is the first attempt to address the challenging light field SOD with the powerful Transformer technique. Extensive experiments on benchmark datasets demonstrate that our models are superior light field SOD approaches and outperform cutting-edge models remarkably.

Teachers’ voices on multimodal input for second or foreign language learning

Multimodal input – which combines written, auditory, and/or visual modalities – is pervasive in everyday life and could serve as a source of rich input in language teaching. In recent years, research has determined that vocabulary learning is one of the clear benefits of being exposed to such input. Regrettably, only a handful of studies have investigated whether and how second language (L2) teachers approach multimodal input in teaching. To further contribute to the research–practice dialogue, we examined factors that influence L2 teachers’ use of multimodal input in L2 teaching. This qualitative case study presents an in-depth analysis of interview data derived from 21 practitioners in various L2 teaching contexts globally. Following three rounds of data analysis, 24 factors were identified and are presented in four themes. The results indicate that teachers: (1) paid close attention to their students’ needs and goals; (2) drew on their own learning and teaching experiences and training supported by research-based practices; (3) relied on sound pedagogical principles; and (4) faced a number of contextual challenges relevant to their curricula and teaching contexts.

Real-time multimodal sensory detection using widefield hippocampal calcium imaging

The hippocampus is a complex structure that has a major role in learning and memory. It also integrates information from multisensory modalities, supporting a comprehensive cognitive map for both spatial and non-spatial information. Previous studies have been limited to real-time spatial decoding, typically using electrodes. However, decoding hippocampal non-spatial information in real time has not been previously described. Here, we have constructed a real-time optical decoder driven by the calcium activity of large neuronal ensembles to decode spatial, visual, and auditory information effectively. Using advanced machine learning techniques, our rapid end-to-end decoding achieves high accuracy and provides a multisensory modality detection method. This method enables the real-time investigation of hippocampal neural coding and allows for direct neural communication with animals and patients affected by functional impairments. The ability to decode multimodal sensory inputs in real time thus forms the basis for an all-optical brain-computer interface.

Wearable brain-computer interface technology and its application

Wearable Brain-Computer Interfaces (BCIs) signify a critical evolution in human-machine communication, driven by the convergence of neuroscience, engineering, and information technology. With applications that span across industrial, medical, and recreational domains, BCIs hold potential to redefine our interaction with the technological landscape. This manuscript elucidates this transformative juncture, bifurcating into passive and active BCIs. In passive BCIs, innovations leveraging Virtual Reality (VR) and Augmented Reality (AR) are delineated, progress has been demonstrated in the classification of passively induced emotional signals, highlighting the emergence of hands-free control systems like quadcopter control and industrial inspections. The discourse on active BCIs unveils advancements such as real-time handwriting and speech decoding, driver drowsiness detection, and emotion recognition, aided by machine learning techniques. Despite groundbreaking progress, challenges in algorithm optimization, adaptability, multimodal signal complexity, and ethics persist. Future directions emphasize the potential of deep learning and multimode input signals collaboration. The manuscript underscores the societal implications, particularly in rehabilitation, communication, and entertainment. The review, therefore, serves as both an appraisal and a roadmap for the burgeoning field of wearable BCIs, underlining its role as a pathway to enhance human capabilities and quality of life.

Second Language Learning Through an Emergent Narrative in a Narrative-Rich Customizable Metaverse Platform

EFL students often struggle with second language (L2) learning, particularly because they learn L2 out of context, which they find unmeaningful and demotivating. To provide a meaningful context for L2 learning and make learning more interesting, the present study utilized a customizable metaverse platform. First, we developed a scenario of a rose theft, and then based on that, we created a town in the metaverse, which offered a variety of multimodal language input and missions to solve the problem and identify the thief. The entire activity was developed grounded in constructivist language learning principles (e.g., contextualized, motivating, interactive, and task-based learning), thus, learners could explore, play, and learn in the playful constructivist space. Seventy-two Korean EFL students participated in the activity. The study employed a mixed method for more robust research. Quantitative data (the pre- and posttests and the post-survey) indicated the students had a great interest in the activity and the learning outcome increased. Qualitative data (the students' journals, the interviews with the teachers and the screen recordings) further evidenced the students' enjoyment of the activity. Particularly, the screen recordings showed how the students participated in the activity in the metaverse, collaboratively created an emergent narrative through interaction, and produced their own narrative to reach the answer. The study suggested pedagogical implications for using the metaverse for L2 learning based on the results of the study.

Compositional Semantic Mix for Domain Adaptation in Point Cloud Segmentation.

Deep-learning models for 3D point cloud semantic segmentation exhibit limited generalization capabilities when trained and tested on data captured with different sensors or in varying environments due to domain shift. Domain adaptation methods can be employed to mitigate this domain shift, for instance, by simulating sensor noise, developing domain-agnostic generators, or training point cloud completion networks. Often, these methods are tailored for range view maps or necessitate multi-modal input. In contrast, domain adaptation in the image domain can be executed through sample mixing, which emphasizes input data manipulation rather than employing distinct adaptation modules. In this study, we introduce compositional semantic mixing for point cloud domain adaptation, representing the first unsupervised domain adaptation technique for point cloud segmentation based on semantic and geometric sample mixing. We present a two-branch symmetric network architecture capable of concurrently processing point clouds from a source domain (e.g.synthetic) and point clouds from a target domain (e.g.real-world). Each branch operates within one domain by integrating selected data fragments from the other domain and utilizing semantic information derived from source labels and target (pseudo) labels. Additionally, our method can leverage a limited number of human point-level annotations (semi-supervised) to further enhance performance. We assess our approach in both synthetic-to-real and real-to-real scenarios using LiDAR datasets and demonstrate that it significantly outperforms state-of-the-art methods in both unsupervised and semi-supervised settings.

&lt;i&gt;Du Iz Tak?&lt;/i&gt;

This paper presents the results of the introduction to a project inspired by the éveil aux langues approach with the storytelling of the picture book Du Iz Tak? by the American/Canadian author Carson Ellis (2016). This propaedeutic activity was a multimodal input given in an unknown language to all children, namely an invented language. The aim of this study is to bring more evidence on how the development of metalinguistic awareness at a young age can help pupils meet their learning needs, especially pupils with SEN. Results show that telling a story in a invented language through the use of multimodal resources to convey meaning not only captures the attention of all pupils in the class, but leads them to observe and manipulate this language by making and developing hypotheses about languages, especially pupils with SEN.

Predicting glaucoma progression using deep learning framework guided by generative algorithm

Glaucoma is a slowly progressing optic neuropathy that may eventually lead to blindness. To help patients receive customized treatment, predicting how quickly the disease will progress is important. Structural assessment using optical coherence tomography (OCT) can be used to visualize glaucomatous optic nerve and retinal damage, while functional visual field (VF) tests can be used to measure the extent of vision loss. However, VF testing is patient-dependent and highly inconsistent, making it difficult to track glaucoma progression. In this work, we developed a multimodal deep learning model comprising a convolutional neural network (CNN) and a long short-term memory (LSTM) network, for glaucoma progression prediction. We used OCT images, VF values, demographic and clinical data of 86 glaucoma patients with five visits over 12 months. The proposed method was used to predict VF changes 12 months after the first visit by combining past multimodal inputs with synthesized future images generated using generative adversarial network (GAN). The patients were classified into two classes based on their VF mean deviation (MD) decline: slow progressors (< 3 dB) and fast progressors (> 3 dB). We showed that our generative model-based novel approach can achieve the best AUC of 0.83 for predicting the progression 6 months earlier. Further, the use of synthetic future images enabled the model to accurately predict the vision loss even earlier (9 months earlier) with an AUC of 0.81, compared to using only structural (AUC = 0.68) or only functional measures (AUC = 0.72). This study provides valuable insights into the potential of using synthetic follow-up OCT images for early detection of glaucoma progression.

Vision Transformer-Based Multilabel Survival Prediction for Oropharynx Cancer After Radiation Therapy

A reliable and comprehensive cancer prognosis model for oropharyngeal cancer (OPC) could better assist in personalizing treatment. In this work, we developed a vision transformer-based (ViT-based) multilabel model with multimodal input to learn complementary information from available pretreatment data and predict multiple associated endpoints for radiation therapy for patients with OPC. A publicly available data set of 512 patients with OPC was used for both model training and evaluation. Planning computed tomography images, primary gross tumor volume masks, and 16 clinical variables representing patient demographics, diagnosis, and treatment were used as inputs. To extract deep image features with global attention, we used a ViT module. Clinical variables were concatenated with the learned image features and fed into fully connected layers to incorporate cross-modality features. To learn the mapping between the features and correlated survival outcomes, including overall survival, local failure-free survival, regional failure-free survival, and distant failure-free survival, we employed 4 multitask logistic regression layers. The proposed model was optimized by combining the multitask logistic regression negative-log likelihood losses of different prediction targets. We employed the C-index and area under the curve metrics to assess the performance of our model for time-to-event prediction and time-specific binary prediction, respectively. Our proposed model outperformed corresponding single-modality and single-label models on all prediction labels, achieving C-indices of 0.773, 0.765, 0.776, and 0.773 for overall survival, local failure-free survival, regional failure-free survival, and distant failure-free survival, respectively. The area under the curve values ranged between 0.799 and 0.844 for different tasks at different time points. Using the medians of predicted risks as the thresholds to identify high-risk and low-risk patient groups, we performed the log-rank test, the results of which showed significantly larger separations in different event-free survivals. We developed the first model capable of predicting multiple labels for OPC simultaneously. Our model demonstrated better prognostic ability for all the prediction targets compared with corresponding single-modality models and single-label models.

The Impact of Multimodal Large Language Models on Health Care’s Future

When large language models (LLMs) were introduced to the public at large in late 2022 with ChatGPT (OpenAI), the interest was unprecedented, with more than 1 billion unique users within 90 days. Until the introduction of Generative Pre-trained Transformer 4 (GPT-4) in March 2023, these LLMs only contained a single mode—text. As medicine is a multimodal discipline, the potential future versions of LLMs that can handle multimodality—meaning that they could interpret and generate not only text but also images, videos, sound, and even comprehensive documents—can be conceptualized as a significant evolution in the field of artificial intelligence (AI). This paper zooms in on the new potential of generative AI, a new form of AI that also includes tools such as LLMs, through the achievement of multimodal inputs of text, images, and speech on health care’s future. We present several futuristic scenarios to illustrate the potential path forward as multimodal LLMs (M-LLMs) could represent the gateway between health care professionals and using AI for medical purposes. It is important to point out, though, that despite the unprecedented potential of generative AI in the form of M-LLMs, the human touch in medicine remains irreplaceable. AI should be seen as a tool that can augment health care professionals rather than replace them. It is also important to consider the human aspects of health care—empathy, understanding, and the doctor-patient relationship—when deploying AI.

PU-NET DEEP LEARNING ARCHITECTURE FOR GLIOMAS BRAIN TUMOUR SEGMENTATION IN MAGNETIC RESONANCE IMAGES

Automatic medical image segmentation is one of the main tasks for many organs and pathology structure delineation. It is also a crucial technique in the posterior clinical examination of brain tumours, like applying radiotherapy or tumour restrictions. Various image segmentation techniques have been proposed and applied to different image types of images. Recently, it has been shown that the deep learning approach accurately segments images, and its implementation is usually straightforward. In this paper, we proposed a novel approach, called PU-net, for automatic brain tumour segmentation in multi-modal magnetic resonance images (MRI) based on deep learning. We introduced an input processing block to a customised fully convolutional network derived from the U-Net network to handle the multi-modal inputs. We performed experiments over the Brats brain tumour dataset collected in 2018 and achieved dice scores of 0.905,0.827, and 0.803 for the whole tumour (WT), tumour core (TC), and enhancing tumour (ET) classes, respectively. This study also provides promising results compared to the traditional machine learning methods, such as support vector machines (SVM), random forest (RF) and other deep learning methods used in this context.