Multimodal Data Streams Research Articles

Advanced Multimodal Sentiment Analysis with Enhanced Contextual Fusion and Robustness (AMSA-ECFR): Symmetry in Feature Integration and Data Alignment

Multimodal sentiment analysis, a significant challenge in artificial intelligence, necessitates the integration of various data modalities for accurate human emotion interpretation. This study introduces the Advanced Multimodal Sentiment Analysis with Enhanced Contextual Fusion and Robustness (AMSA-ECFR) framework, addressing the critical challenge of data sparsity in multimodal sentiment analysis. The main components of the proposed approach include a Transformer-based model employing BERT for deep semantic analysis of textual data, coupled with a Long Short-Term Memory (LSTM) network for encoding temporal acoustic features. Innovations in AMSA-ECFR encompass advanced feature encoding for temporal dynamics and an adaptive attention-based model for efficient cross-modal integration, achieving symmetry in the fusion and alignment of asynchronous multimodal data streams. Additionally, the framework employs generative models for intelligent approximation of missing features. It ensures robust alignment of high-level features with multimodal data context, effectively tackling issues of incomplete or noisy inputs. In simulation studies, the AMSA-ECFR model demonstrated superior performance against existing approaches. The symmetrical approach to feature integration and data alignment contributed significantly to the model’s robustness and precision. In simulations, the AMSA-ECFR model demonstrated a 10% higher accuracy and a 15% lower mean absolute error than the current best multimodal sentiment analysis frameworks.

Insights into ALD and AUD diagnosis and prognosis: Exploring AI and multimodal data streams.

The rapid evolution of artificial intelligence and the widespread embrace of digital technologies have ushered in a new era of clinical research and practice in hepatology. Although its potential is far from realization, these significant strides have generated new opportunities to address existing gaps in the delivery of care for patients with liver disease. In this review, we discuss how artificial intelligence and opportunities for multimodal data integration can improve the diagnosis, prognosis, and management of alcohol-associated liver disease. An emphasis is made on how these approaches will also benefit the detection and management of alcohol use disorder. Our discussion encompasses challenges and limitations, concluding with a glimpse into the promising future of these advancements.

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

AbstractPerception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter.

A feasibility study for a unified, multimodal analysis of online information foraging in health-related topics.

Digital health literacy (DHL) is the ability to find, understand, and appraise online health-related information, as well as apply it to health behavior. It has become a core competence for navigating online information and health service environments. DHL involves solving ill-structured problems, where the problem and its solution are not clearcut and may have no single answer, such as in the process of sensemaking. We employ and expand on information foraging theory to address how experts and novices in information retrieval perform a search task. Our overarching aim is to pinpoint best practices and pitfalls in understanding and appraising health-related information online to develop a digital intervention to increase DHL and critical thinking. In this feasibility study, we recruited a total of twenty participants for our expert and novice subsamples. We collected sociodemographic data with a self-developed survey, video data through an observation protocol of a 10-minute search task, as well as audio-video data via a retrospective think-aloud. The three, multimodal data streams were transcribed and aligned. Codes were developed inductively in several iterations, then applied deductively to the entire dataset. Tabularized, coded and segmented qualitative data were used to create various quantitative models, which demonstrate viability for the qualitative and statistical comparison of our two subsamples. Data were visualized with Epistemic Network Analysis to analyze code co-occurrences in the three aligned data streams, and with Qualitative/Unified Exploration of State Transitions to examine the order in which participants in our two subsamples encountered online content. This paper describes our methods and planned analyses elaborated with mock figures. Quantifying qualitative data, aligning data streams, and representing all information in a tabularized dataset allows us to group data according to various participant attributes and employ data visualization techniques to pinpoint patterns therein.

A method for synchronized use of EEG and eye tracking in fully immersive VR.

This study explores the synchronization of multimodal physiological data streams, in particular, the integration of electroencephalography (EEG) with a virtual reality (VR) headset featuring eye-tracking capabilities. A potential use case for the synchronized data streams is demonstrated by implementing a hybrid steady-state visually evoked potential (SSVEP) based brain-computer interface (BCI) speller within a fully immersive VR environment. The hardware latency analysis reveals an average offset of 36 ms between EEG and eye-tracking data streams and a mean jitter of 5.76 ms. The study further presents a proof of concept brain-computer interface (BCI) speller in VR, showcasing its potential for real-world applications. The findings highlight the feasibility of combining commercial EEG and VR technologies for neuroscientific research and open new avenues for studying brain activity in ecologically valid VR environments. Future research could focus on refining the synchronization methods and exploring applications in various contexts, such as learning and social interactions.

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

There is a wide application of deep learning technique to unimodal medical image analysis with significant classification accuracy performance observed. However, real-world diagnosis of some chronic diseases such as breast cancer often require multimodal data streams with different modalities of visual and textual content. Mammography, magnetic resonance imaging (MRI) and image-guided breast biopsy represent a few of multimodal visual streams considered by physicians in isolating cases of breast cancer. Unfortunately, most studies applying deep learning techniques to solving classification problems in digital breast images have often narrowed their study to unimodal samples. This is understood considering the challenging nature of multimodal image abnormality classification where the fusion of high dimension heterogeneous features learned needs to be projected into a common representation space. This paper presents a novel deep learning approach combining a dual/twin convolutional neural network (TwinCNN) framework to address the challenge of breast cancer image classification from multi-modalities. First, modality-based feature learning was achieved by extracting both low and high levels features using the networks embedded with TwinCNN. Secondly, to address the notorious problem of high dimensionality associated with the extracted features, binary optimization method is adapted to effectively eliminate non-discriminant features in the search space. Furthermore, a novel method for feature fusion is applied to computationally leverage the ground-truth and predicted labels for each sample to enable multimodality classification. To evaluate the proposed method, digital mammography images and digital histopathology breast biopsy samples from benchmark datasets namely MIAS and BreakHis respectively. Experimental results obtained showed that the classification accuracy and area under the curve (AUC) for the single modalities yielded 0.755 and 0.861871 for histology, and 0.791 and 0.638 for mammography. Furthermore, the study investigated classification accuracy resulting from the fused feature method, and the result obtained showed that 0.977, 0.913, and 0.667 for histology, mammography, and multimodality respectively. The findings from the study confirmed that multimodal image classification based on combination of image features and predicted label improves performance. In addition, the contribution of the study shows that feature dimensionality reduction based on binary optimizer supports the elimination of non-discriminant features capable of bottle-necking the classifier.

Discrete Emotion Synchronicity and Video Engagement on Social Media: A Moment-to-Moment Analysis

ABSTRACT Although previous studies have identified several discrete emotion features that drive video diffusion on social media, little research has viewed the video consumption experience as a continuous communication process and combined viewer-side moment-to-moment (MTM) emotional reactions with supply-side MTM emotional dynamics. In this article, we examine how discrete emotion synchronicity, which refers to the synchronicity between MTM discrete emotions in the video content and MTM viewer emotional reactions, is associated with user engagement on video-based social media. Using a set of state-of-the-art deep learning algorithms and a regression model, we first extract discrete emotion synchronicity from videos with multimodal data streams. Then, we conduct an empirical analysis to determine the impact of discrete emotion synchronicity on user engagement. Our results indicate that discrete emotion synchronicity does not always lead to higher user engagement on video-based social media. First, discrete emotion synchronicity that increases the number of video tips (i.e., joy) cannot increase the number of saves and shares. Second, discrete emotion synchronicity (i.e., calm, solemnity, and sadness) has a negative effect on 3 types of engagement behaviors in some cases. Third, the negative effect of discrete emotion synchronicity outweighs its positive effect. Our work offers new insight into how discrete emotions facilitate or hinder user engagement in the context of video-based social media and has practical implications for firms’ marketing strategies on video-oriented social media.

A survey of multimodal hybrid deep learning for computer vision: Architectures, applications, trends, and challenges

In recent years, deep learning algorithms have rapidly revolutionized artificial intelligence, particularly machine learning, enabling researchers and practitioners to extend previously hand-crafted feature extraction procedures. In particular, deep learning uses adaptive learning processes to learn more complex and informative patterns from datasets of varying sizes. With the increasing availability of multimodal data streams and recent advances in deep learning algorithms, multimodal deep learning is on the rise. This requires the development of complex models that can process and analyze multimodal information in a consistent manner. However, unstructured data can come in many different forms (also known as modalities). Extracting relevant features from this data remains an ambitious goal for deep learning researchers. According to the literature, most deep learning systems consist of a single architecture (i.e., standalone deep learning). When two or more deep learning architectures are combined over multiple sensory modalities, the result is called a multimodal hybrid deep learning model. Since this research direction has received much attention in the field of deep learning, the purpose of this survey is to provide a broader overview of the topic. In this paper, we provide a comprehensive review of recent advances in multimodal hybrid deep learning, including a thorough analysis of the most commonly developed hybrid architectures. In particular, one of the main challenges in multimodal hybrid analysis is the ability of these architectures to systematically integrate cross-modal features in hybrid designs. Therefore, we propose a generic framework for multimodal hybrid learning that focuses mainly on fusion methods. We also identify trends and challenges in multimodal hybrid learning and provide insights and directions for future research. Our findings show that multimodal hybrid learning can perform well in a variety of challenging computer vision applications and tasks.

A multimodal digital biomarker of functional deficits in early‐stage Alzheimer’s disease: results of the RADAR‐AD study

AbstractBackgroundRemote monitoring technologies (RMTs), such as smartphone apps, smartwatches, and in‐home sensors, are rapidly changing the way functional and cognitive performance is measured in Alzheimer’s disease (AD) patients. Here, we present results from the European RADAR‐AD study on the use of multimodal data streams for the identification of functional deficits across all syndromic stages of AD.MethodFour study groups (Healthy controls (HC), preclinical AD (pre. AD), prodromal AD (pro. AD), and mild AD) were included in this cross‐sectional study. The RMT features (gait measures from Timed up and Go (TUG), Dual Task Effect (DTE) using physilog, acoustic features from Speech task in Mezurio, neurocognitive function using Altoida, managing finances with Banking app) with in‐clinic neuropsychological (NP) tests, activities of daily living with Amsterdam IADL and demographics (age, gender, education years) were analyzed for different combinations of the multimodal digital biomarker of disease stage in AD across different pairwise comparisons. The analysis includes data from 175 participants (HC = 67, Pre.AD = 26, Pro.AD = 50, Mild AD = 32) collected for 8 weeks. An extreme gradient boosting (XGBoost) machine learning model was trained to obtain a multimodal biomarker of AD disease stage with repeated cross‐validation (5‐fold with 4 repeats) (Figure 1). This investigation is part of the ongoing RADAR‐AD study.ResultThe multimodal combination of RMTs achieved a mean AUC of > 0.60 in all pairwise comparisons with a mean AUC > 0.65 for HC vs (Pre.AD, Pro.AD and Mild) (Figures 2 and 3). The addition of NP tests increases the performance considerably across all pairwise comparisons except HC vs Pre.AD.ConclusionOur results highlight the advantage of combining RMTs to identify functional deficits in the early stage of AD. In particular, in prodromal and mild AD patients, a combined signal shows much more strength compared to individual tests. This work has received support from the EU/EFPIA Innovative Medicines Initiative Joint Undertaking (grant No 806999). www.imi.europa.eu. This communication reflects the views of the RADAR‐AD consortium and neither IMI nor the European Union and EFPIA are liable for any use that may be made of the information contained herein.

PATCH

Recent advancements in deep learning have shown that multimodal inference can be particularly useful in tasks like autonomous driving, human health, and production line monitoring. However, deploying state-of-the-art multimodal models in distributed IoT systems poses unique challenges since the sensor data from low-cost edge devices can get corrupted, lost, or delayed before reaching the cloud. These problems are magnified in the presence of asymmetric data generation rates from different sensor modalities, wireless network dynamics, or unpredictable sensor behavior, leading to either increased latency or degradation in inference accuracy, which could affect the normal operation of the system with severe consequences like human injury or car accident. In this paper, we propose PATCH, a framework of speculative inference to adapt to these complex scenarios. PATCH serves as a plug-in module in the existing multimodal models, and it enables speculative inference of these off-the-shelf deep learning models. PATCH consists of 1) a Masked-AutoEncoder-based cross-modality imputation module to impute missing data using partially-available sensor data, 2) a lightweight feature pair ranking module that effectively limits the searching space for the optimal imputation configuration with low computation overhead, and 3) a data alignment module that aligns multimodal heterogeneous data streams without using accurate timestamp or external synchronization mechanisms. We implement PATCH in nine popular multimodal models using five public datasets and one self-collected dataset. The experimental results show that PATCH achieves up to 13% mean accuracy improvement over the state-of-art method while only using 10% of training data and reducing the training overhead by 73% compared to the original cost of retraining the model.

A Cross-Modal Dynamic Attention Neural Architecture to Detect Anomalies in Data Streams from Smart Communication Environments

Detecting anomalies in data streams from smart communication environments is a challenging problem that can benefit from novel learning techniques. The Attention Mechanism is a very promising architecture for addressing this problem. It allows the model to focus on specific parts of the input data when processing it, improving its ability to understand the meaning of specific parts in context and make more accurate predictions. This paper presents a Cross-Modal Dynamic Attention Neural Architecture (CM-DANA) by expanding on state-of-the-art techniques. It is a novel dynamic attention mechanism that can be trained end-to-end along with the rest of the model using multimodal data streams. The attention mechanism calculates attention weights for each position in the input data based on the model’s current state by a hybrid method called Cross-Modal Attention. Specifically, the proposed model uses multimodal learning tasks where the input data comes from different cyber modalities. It combines the relevant input data using these weights to produce an attention vector in order to detect suspicious abnormal behavior. We demonstrate the effectiveness of our approach on a cyber security anomalies detection task using multiple data streams from smart communication environments.

A feasibility study for a unified, multimodal analysis of online information foraging in health-related topics

Background: Digital health literacy (DHL) is the ability to find, understand, and appraise online health-related information, as well as apply it to health behavior. It has become a core competence for navigating online information and health service environments. DHL involves solving ill-structured problems, where the problem and its solution are not clearcut and may have no single answer, such as in the process of sensemaking. We employ and expand on information foraging theory to address how experts and novices in information retrieval perform a search task. Our overarching aim is to pinpoint best practices and pitfalls in understanding and appraising health-related information online to develop a digital intervention to increase DHL and critical thinking. Methods: In this feasibility study, we recruited a total of twenty participants for our expert and novice subsamples. We collected sociodemographic data with a self-developed survey, video data through an observation protocol of a 10-minute search task, as well as audio-video data via a retrospective think-aloud. The three, multimodal data streams were transcribed and aligned. Codes were developed inductively in several iterations, then applied deductively to the entire dataset. Tabularized, coded and segmented qualitative data were used to create various quantitative models, which demonstrate viability for the qualitative and statistical comparison of our two subsamples. Results: Data were visualized with Epistemic Network Analysis to analyze code co-occurrences in the three aligned data streams, and with Qualitative/Unified Exploration of State Transitions to examine the order in which participants in our two subsamples encountered online content. Conclusions: This paper describes our methods and planned analyses elaborated with mock figures. Quantifying qualitative data, aligning data streams, and representing all information in a tabularized dataset allows us to group data according to various participant attributes and employ data visualization techniques to pinpoint patterns therein.

Disembodied AI and the limits to machine understanding of students' embodied interactions.

The embodiment turn in the Learning Sciences has fueled growth of multimodal learning analytics to understand embodied interactions and make consequential educational decisions about students more rapidly, more accurately, and more personalized than ever before. Managing demands of complexity and speed is leading to growing reliance by education systems on disembodied artificial intelligence (dAI) programs, which, ironically, are inherently incapable of interpreting students' embodied interactions. This is fueling a potential crisis of complexity. Augmented intelligence systems offer promising avenues for managing this crisis by integrating the strengths of omnipresent dAI to detect complex patterns of student behavior from multimodal datastreams, with the strengths of humans to meaningfully interpret embodied interactions in service of consequential decision making to achieve a balance between complexity, interpretability, and accountability for allocating education resources to children.

Contextually aware roadside radiation measurement testbed

We demonstrate a contextually aware multimodal roadside radiation measurement detection testbed for traffic monitoring applications in nuclear nonproliferation. Many variables in traffic such as vehicle or cargo size, mass, speed, shape, and distance of closest approach can have significant impacts on the radiation measured from a vehicle-transported radiation source. These factors can lead to uncertainties in the analysis of the radiation source, especially for lower-strength radiation sources of interest. Our testbed, known as the Multimodal Measurement System (MMS) uses non-radiation sensors including magnetometers, geophones, radiofrequency receivers, cameras, and LiDAR to extract contextual information about vehicles passing by the system. These contextual data can then be fused with data from radiation measurements to increase the system’s sensitivity and accuracy in nuclear threat detection applications. This work describes the instrumentation of the MMS and its data acquisition pipeline. Furthermore, we describe the pre-analysis performed on the raw multimodal data streams for data fusion, and the high-level machine learning analyses for detection and characterization. The variety of sensors within the MMS provides a valuable testbed that can be used to identify the combinations of contextual sensors that provide the greatest improvements to radiation source detection and characterization within the restrictions for various proliferation detection applications. The MMS is also modular so that additional combinations of sensors can be explored in the future.

ARGUS: Visualization of AI-Assisted Task Guidance in AR.

The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi-year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine-tune the components of the AR assistant.

Automated Power System Fault Prediction and Precursor Discovery Using Multi-Modal Data

Electric power system operators monitor large multi-modal data streams from wide service areas. The current data setups stand to get more complex as utilities add more smart-grid sensors to collect additional data from power system substations and other in-situ locations. We propose a methodology to utilize multi-modal data for automated power system fault prediction, and precursor discovery that takes advantage of not only the utility owned measurements but also an abundance of data from other related databases such as weather observation systems. The process is automated to help operators analyze multi-modal data that may be impossible to process manually due to the size and variety. We automatically preprocess multi-source data and learn a joint latent representation from collocated streamed, sparse, and high-dimensional data collected from Phasor Measurement Units and external weather data. Then we utilize multi-instance learning to predict events and discover precursors simultaneously without relying on post-mortem studies of fault signatures. We apply the proposed methodology to provide early predictions of faults in the U.S. Western Interconnection. AU-ROC of 0.94 is achieved in predicting events by utilizing information 5 hours before event time using season-specific models. We show how precursors can be extracted from multi-modal data and interpreted for predicted events.

基于无线传感器网络的农业温室环境的低功耗监测方法

In order to improve the reliability of wireless sensor networks and reduce the power consumption in the monitoring process, the low power consumption monitoring method of agricultural greenhouse environment based on wireless sensor networks is studied. The terminal node of wireless sensor network is constructed by using CO2 sensor, temperature and humidity sensor and illumination sensor. In the sensor network layout stage, in order to reduce the node power consumption, considering the number of nodes and network coverage, the cuckoo search algorithm is used to optimize the node layout model. The communication module uses LEACH protocol to transmit the environmental data collected by the terminal node to the monitoring center. The cluster head link is selected in LEACH protocol to save energy. After receiving the environmental data collected by the terminal node, the monitoring center uses multi-dimensional data to identify and locate abnormal environmental data according to the correlation between multi-modal data streams in the same node, so as to realize the environmental monitoring of agricultural greenhouse. The experimental results show that the data transmission of the research method is stable and reliable, and the node’s power consumption in the process of internal environment monitoring of the research object is effectively reduced.

Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data

Learning sciences are embracing the significant role technologies can play to better detect, diagnose, and act upon self-regulated learning (SRL). The field of SRL is challenged with the measurement of SRL processes to advance our understanding of how multimodal data can unobtrusively capture learners' cognitive, metacognitive, affective, and motivational states over time, tasks, domains, and contexts. This paper introduces a self-regulated learning processes, multimodal data, and analysis (SMA) grid and maps joint and individual research of the authors (63 papers) over the last five years onto the grid. This shows how multimodal data streams were used to investigate SRL processes. The two-dimensional space on the SMA grid is helpful for visualizing the relations and possible combinations between the data streams and how the measurement of SRL processes. This overview serves as an analytical introduction to the current special issue “Advancing SRL Research with Artificial Intelligence (AI)” and we encourage to position new research and unexplored frontiers. We emphasize the need for intensive and strategic collaboration to accelerate progress using new interdisciplinary methods to develop accurate measurement of SRL in educational technologies.

Low-Latency Speculative Inference on Distributed Multi-Modal Data Streams

While multi-modal deep learning is useful in distributed sensing tasks like human tracking, activity recognition, and audio and video analysis, deploying state-of-the-art multi-modal models in a wirelessly networked sensor system poses unique challenges. The data sizes for different modalities can be highly asymmetric (e.g., video vs. audio), and these differences can lead to significant delays between streams in the presence of wireless dynamics. Therefore, a slow stream can significantly slow down a multimodal inference system in the cloud, leading to either increased latency (when blocked by the slow stream) or degradation in inference accuracy (if inference proceeds without waiting).

Socially CompliAnt Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation

Social navigation is the capability of an autonomous agent, such as a robot, to navigate in a “socially compliant” manner in the presence of other intelligent agents such as humans. With the emergence of autonomously navigating mobile robots in human-populated environments (e.g., domestic service robots in homes and restaurants and food delivery robots on public sidewalks), incorporating socially compliant navigation behaviors on these robots becomes critical to ensuring safe and comfortable human-robot coexistence. To address this challenge, imitation learning is a promising framework, since it is easier for humans to demonstrate the task of social navigation rather than to formulate reward functions that accurately capture the complex multi-objective setting of social navigation. The use of imitation learning and inverse reinforcement learning to social navigation for mobile robots, however, is currently hindered by a lack of large-scale datasets that capture socially compliant robot navigation demonstrations in the wild. To fill this gap, we introduce Socially CompliAnt Navigation Dataset ( SCAND )–a large-scale, first-person-view dataset of socially compliant navigation demonstrations. Our dataset contains 8.7 hours, 138 trajectories, 25 miles of socially compliant, human tele-operated driving demonstrations that comprises multi-modal data streams including 3D lidar, joystick commands, odometry, visual and inertial information, collected on two morphologically different mobile robots–a Boston Dynamics Spot and a Clearpath Jackal–by four different human demonstrators in both indoor and outdoor environments. We additionally perform preliminary analysis and validation through real-world robot experiments and show that navigation policies learned by imitation learning on SCAND generate socially compliant behaviors.