Multimodal State Research Articles

Explainable Feature Engineering for Multi-Modal Tissue State Monitoring Based on Impedance Spectroscopy.

One of the most promising approaches to food quality assessments is the use of impedance spectroscopy combined with machine learning. Thereby, feature selection is decisive for a high classification accuracy. Physically based features have particularly significant advantages because they are able to consider prior knowledge and to concentrate the data into pertinent understandable information, building a solid basis for classification. In this study, we aim to identify physically based measurable features for muscle type and freshness classifications of bovine meat based on impedance spectroscopy measurements. We carry out a combined study where features are ranked based on their F1-score, cumulative feature selection, and t-distributed Stochastic Neighbor Embedding (t-SNE). In terms of features, we analyze the characteristic points (CPs) of the impedance spectrum and the model parameters (MPs) obtained by fitting a physical model to the measurements. The results show that either MPs or CPs alone are sufficient for detecting muscle type. Combining capacitance (C) and extracellular resistance (Rex) or the modulus of the characteristic point Z1 and the phase at the characteristic frequency of the beta dispersion (Phi2) leads to accurate separation. In contrast, the detection of freshness is more challenging. It requires more distinct features. We achieved a 90% freshness separation using the MPs describing intracellular resistance (Rin) and capacitance (C). A 95.5% freshness separation was achieved by considering the phase at the end of the beta dispersion (Phi3) and Rin. Including additional features related to muscle type improves the separability of samples; ultimately, a 99.6% separation can be achieved by selecting the appropriate features.

(Invited) Multi-Modal Solid-State Gas Sensors-Based Breath Analysis System with Convolutional Neural Network (CNN) for Early Detection of Lung Cancer

Currently, digital healthcare technologies based on Internet of Things (IOT) are emerging, and the market for digital healthcare was estimated to be 151.5 billion dollars by 2020. Among the digital healthcare technologies, disease diagnosis is attracting maximum attention. Traditional diagnostic methods, such as radiology and human biological material analysis, have limitations of long study time, high cost, radiation exposure, and pain. Therefore, non-invasive diagnostics methods, for early diagnosis of specific diseases and personal healthcare management, have attracted attention.Breath sensors are one of the promising candidates for the realization of non-invasive diagnosis owing to their ease of use, safety, and cost effectiveness. The exhaled breath analysis technique holds great promise as a non-invasive method for diagnosing lung cancer. In the breath of lung cancer patients, various volatile organic compounds (VOCs) markers are generated during metabolic processes. Analyzing these VOCs markers presents a compelling avenue for early and non-invasive lung cancer diagnosis. Solid state gas sensors capable of measuring specific VOCs are currently being researched and utilized as commercial gas sensors for this purpose. However, the concentration of lung cancer VOCs markers in exhaled breath is commonly very low, and individual commercial gas sensors often lack the necessary sensitivity and selectivity to accurately diagnose lung cancer. In this study, we propose a multimodal solid state electronic biosensor system designed to differentiate lung cancer VOCs characteristics among multiple VOCs, enabling exhalation-based lung cancer diagnosis. Initially, we developed multimodal biosensors using more than 20 gas sensors, consisting of metal oxide semiconductor sensors, electrochemical sensors, photoionized detectors, metalorganic frame sensors and so on. And each exhibiting varying sensitivity profiles for complex gas mixtures. Additionally, we designed a classification algorithm based on convolutional neural networks (CNN) and multi-layered perceptron (MLP) to distinguish between lung cancer patients and healthy individuals using the output from these diverse multimodal gas sensors. To evaluate the system's performance, we collected exhaled air samples from 74 lung cancer patients and 107 normal subjects, conducting clinical trials with our developed system. We did model evaluation with confusion matrix and ROC curve. The results of these tests confirmed that our system achieved a high predictive accuracy of over 95% in classifying lung cancer patients. The simplicity of the sensor configuration in our developed system makes it suitable for large-scale screening tests to identify potential candidates for precise lung cancer diagnosis. This technology has the potential to contribute to reduced national welfare costs and improved patient survival rates through early diagnosis.

Multimodal rapid identification of growth stages and discrimination of growth status for Morchella

We introduce a multimodal rapid identification and growth status discrimination method for morchella. Based on the unique biological characteristics and growth environmental requirements of morchella, the efficient and accurate identification of key growth stages of morchella is achieved through the integration of multimodal information acquisition technology. During the rapid identification process of the growth stage of Morchella, the Multi Stage Vision Enhanced Position Encoding Vision Transformer (MS-EP ViT) model is adopted. By introducing multi-stage input embedding, enhanced position encoding, and optimized Transformer Encoder layers, the performance of the model in identifying different growth stages of Morchella mushrooms is significantly improved. In the multimodal Morchella growth state discrimination method, text and image modalities are integrated, a Non downsampled Contourlet Transform Mask Region based Convolutional Neural Network (NSCT Mask R-CNN) model is designed, and a multimodal feature extraction strategy combining Non downsampled Contourlet Transform (NSCT) features with environmental features is explored. This strategy effectively achieves the goals of object detection and instance segmentation, and thus we have accurately evaluated the growth status of Morchella in the later stages of mulberry, young mushroom, and mature. The experimental results show that both models have achieved significant improvements in recognition accuracy and stability, and the rationality of hyperparameter settings has been verified through convergence and parameter sensitivity experiments. Overall, we provide a more accurate and efficient identification method for monitoring the growth of Morchella, which helps to better understand the growth of Morchella and provides scientific basis for optimizing its growth environment.

Deep reinforcement learning navigation via decision transformer in autonomous driving.

In real-world scenarios, making navigation decisions for autonomous driving involves a sequential set of steps. These judgments are made based on partial observations of the environment, while the underlying model of the environment remains unknown. A prevalent method for resolving such issues is reinforcement learning, in which the agent acquires knowledge through a succession of rewards in addition to fragmentary and noisy observations. This study introduces an algorithm named deep reinforcement learning navigation via decision transformer (DRLNDT) to address the challenge of enhancing the decision-making capabilities of autonomous vehicles operating in partially observable urban environments. The DRLNDT framework is built around the Soft Actor-Critic (SAC) algorithm. DRLNDT utilizes Transformer neural networks to effectively model the temporal dependencies in observations and actions. This approach aids in mitigating judgment errors that may arise due to sensor noise or occlusion within a given state. The process of extracting latent vectors from high-quality images involves the utilization of a variational autoencoder (VAE). This technique effectively reduces the dimensionality of the state space, resulting in enhanced training efficiency. The multimodal state space consists of vector states, including velocity and position, which the vehicle's intrinsic sensors can readily obtain. Additionally, latent vectors derived from high-quality images are incorporated to facilitate the Agent's assessment of the present trajectory. Experiments demonstrate that DRLNDT may achieve a superior optimal policy without prior knowledge of the environment, detailed maps, or routing assistance, surpassing the baseline technique and other policy methods that lack historical data.

Typical cases of online teaching quality evaluation based on multimodal affective state analysis

Typical cases of online teaching quality evaluation based on multimodal affective state analysis

A Multimodal Robust Simultaneous Localization and Mapping Approach Driven by Geodesic Coordinates for Coal Mine Mobile Robots

Mobile robots in complex underground coal mine environments are still unable to achieve accurate pose estimation and the real-time reconstruction of scenes with absolute geographic information. Integrated terrestrial-underground localization and mapping technologies have still not been effectively developed. This paper proposes a multimodal robust SLAM method based on wireless beacon-assisted geographic information transmission and lidar-IMU-UWB elastic fusion mechanism (LIU-SLAM). In order to obtain the pose estimation and scene models consistent with the geographic information, the construction of two kinds of absolute geographic information constraints based on UWB beacons is proposed. An elastic multimodal fusion state estimation mechanism is designed based on incremental factor graph optimization. A tightly coupled lidar-inertial odometry is firstly designed to construct the lidar-inertial local transformation constraints, which are further integrated with the absolute geographic constraints by UWB anchors through a loosely coupled approach. Extensive field tests based on coal mine robots have been conducted in scenarios such as underground garages and underground coal mine laneways. The results show that the proposed geodesic-coordinate driven multimode robust SLAM method can obtain absolute localization accuracy within 25 cm with practical robustness and real-time performance in different underground application scenarios. The wireless beacon-assisted geodesic-coordinate transmission strategy can provide a plug-and-play customized solution for precise positioning and scene modeling in complex scenarios of various coal mine robot application.

Tuned mass damper on spacecraft reaction wheel assembly

Considering the wide frequency range and time-varying characteristics of reaction wheel assembly (RWA) micro-vibrations, an embedded tuned mass damper (TMD) was proposed herein to reduce the influence of the RWA micro-vibration effect on spacecraft. First, dynamics model of the RWA and a mechanical model of the angular contact ball bearing were constructed, and the time-varying characteristics of the RWA were theoretically analyzed. Based on single- and multiple-degree-of-freedom vibration absorption dynamics models, it was shown that the TMD could be applied to the vibration suppression of the RWA’s multi-modal and time-varying state by changing the design parameters of the TMD. Based on the RWA single-degree-of-freedom equivalent dynamics model, the effects of the TMD natural frequency and damping on the displacement amplitude ratio of the RWA were analyzed, and the sensitivities of the RWA resonance peak amplitude and resonance frequency to the two parameters were analyzed by the Sobol sensitivity analysis method. Then, according to the analysis results, the structural design of the TMD in the RWA was guided, and a test platform was built to obtain the RWA micro-vibration data. Finally, a normalized amplitude-frequency response identification algorithm was used to eliminate the effects of non-linearities in the experimental data and to identify the RWA dynamics in a time-varying state, thus obtaining the RWA amplitude–frequency response curve and verifying the results of the theoretical analysis. Theoretical and experimental data showed that the addition of a TMD structure with variable damping and stiffness within the RWA allowed for micro-vibration suppression for arbitrary modes and improved the RWA vibration characteristics. This could provide convenience for different specifications of RWA vibration suppression as well as reference for further research on the effective application of TMDs in RWAs.

Late integration of vision and proprioception during perturbed reaches.

The motor system corrects rapidly, but selectively, for perturbations to ongoing reaching movements, depending on the constraints of the task. To account for such sophistication, it has been postulated that corrections are based on an estimated limb state that integrates all sensory changes caused by the perturbation, taking into account their processing delays. Here, we asked if information from different sensory modalities is integrated immediately or processed separately in the early phase of a response. We perturbed the estimated state of the limb with both unimodal and bimodal visual and proprioceptive perturbations without changing the actual limb state. For visual perturbations, a cursor representing the hand was shifted to the left or the right relative to the true hand location. For proprioceptive perturbations, the biceps or triceps muscles were vibrated, which induced illusory limb-state changes to the right or the left. In the bimodal condition, the perturbations to vision and proprioception were either congruent or incongruent in their directions. Response latencies show that it takes ∼100 ms longer to respond to unimodal visual perturbations than to unimodal proprioceptive perturbations. Responses to bimodal perturbations show that it takes an additional ∼100 ms beyond the response to unimodal visual perturbations for intermodal consistency to impact the response. These results suggest that visual and proprioceptive signals are initially processed separately for state estimation and only combined at the level of the limb's motor output, instead of being immediately integrated into a single state estimate of the limb.NEW & NOTEWORTHY Both visual and proprioceptive signals provide information about arm state during reaching. By perturbing the perceived, but not the actual, position of the hand in both modalities using visual disturbances and muscle vibration, we examined multimodal integration and state estimation during reaching. Our results suggest that the early reach corrections are based on separate state estimates from the two sensory modalities and only later are based on a combined state estimate.

Wave climate and energy resources in American Samoa from a 42-year high-resolution hindcast

This paper presents an analysis of the wave climate and a characterization of the wave energy resources in American Samoa, a U.S. territory covering seven south central Pacific islands and atolls. A numerical wave model based on WAVEWATCH III® and unstructured SWAN was developed, validated, and executed for 1979–2020 to generate a hindcast dataset suitable for resource characterization. Model-data comparisons were performed with measurements collected in situ and from satellite-based altimeters. The model was found to perform well with a bias in significant wave height of −0.14 m and −0.06 m against buoy and altimeters, respectively. The multimodal sea state of American Samoa is investigated by identifying the sources of energy reaching the islands and partitioning the wave spectrum accordingly. The wave resources characterization follows the International Electrotechnical Commission Technical Specifications to be compatible with studies performed for other U.S. regions. The average omnidirectional wave power at 2 km from shore around Tutuila, the main island of American Samoa, is 14 kW/m. Locations east and west of the islands have a more consistent resource throughout the year because the northwest and southwest swells that are dominant during different seasons complement each other.

Multimodal Assessment of Changes in Physiological Indicators when Presenting a Video Fragment on Screen (2D) versus a VR (3D) Environment.

The increasing role of virtual environments in society, especially in the context of the pandemic and evolving metaverse technologies, requires a closer study of the physiological state of humans using virtual reality (VR) for entertainment, work, or learning. Despite the fact that many physiological reactions to the content presented in various modalities under VR conditions have already been described, often these studies do not reflect the full range of changes in the physiological reactions that occur to a person during their immersion in the virtual world. This study was designed to find and compare the most sensitive physiological indicators that change when viewing an emotionally intense video fragment in standard format on screen and in virtual reality conditions (in a VR helmet). The research methodology involved randomly presenting a group of subjects with visual content-a short video clip-first on screen (2D) and then in a virtual reality helmet (3D). A special feature of this study is the use of multimodal physiological state assessment throughout the content presentation, in conjunction with psychological testing of the study participants before and after the start of the study. It has been discovered that the most informative physiological indicators reflecting the subjects' condition under virtual reality conditions were changes in theta rhythm amplitude, skin conductance, standard deviation of normal RR-intervals (SDRR), and changes in photoplethysmogram (PPG). The study results suggest that in the process of immersion in a virtual environment, the participants develop a complex functional state, different from the state when watching on screen, which is characterised by the restructuring of autonomic regulation and activation of emotion structures of the brain.

Pattern-adaptive generative adversarial network with sparse data for traffic state estimation

Accurate traffic state estimation is vital basis for traffic control and management applications. However, owing to the multi-modality of traffic state patterns, the estimation methods are hard to adapt to the state variation over the urban roads and time periods. Especially, the sparse sampling of traffic state makes the traffic pattern recognition easy to bias and increases uncertainty in the estimation. To address the problem of multi-modal traffic state estimation under sparse data, we propose a pattern-adaptive generative adversarial network, named PA-GAN. In the PA-GAN, the Bayesian Inference is introduced to place multi-modal posterior distributions over the network parameters. Therefore, to targeted state estimation for each pattern, the posterior sampling will adaptively activate the corresponding parameters of each traffic pattern according to the context features. Then, the PA-GAN learns traffic patterns from sparse sampling data by the traffic state generator and the discriminator, in which an error-feedback mechanism uses multi-level traffic features to correct the estimation under an encoder–decoder framework. To evaluate the proposed PA-GAN, we use two real-world datasets to conduct comprehensive case studies about multi-modal traffic patterns and sparse sampling. The experimental results demonstrate that the PA-GAN can outperform other estimation methods and that the Bayesian Inference can improve the adaptability ability of the learning network to various traffic states.

Multimodal Opera Performance Form Based on Human-Computer Interaction Technology

“Audience Engagement (AE)” describes how a stage performance affects the audience’s thoughts, provokes a bodily response, and spurs cognitive growth. With little audience involvement, theatre performing arts like opera typically have difficulty keeping audiences’ attention. The brain-computer interaction (BCI) technology could be used in opera performances to alter the audience’s emotional experience. Nevertheless, for such BCI systems to function, they must accurately identify their participants’ present emotional states. Although difficult to evaluate, audience participation is a vital sign of how well an opera performs. Practical methodological approaches for real-time perception and comprehension of audience emotions include psychological and physiological assessments. Hence, a multimodal emotional state detection technique (MESDT) for enhancing the AE in opera performance using BCI has been proposed. Three essential steps make up a conceptual MESDT architecture. An electroencephalogram (EEG) and other biological signs from the audience are first captured. Second, the acquired signals are processed, and the BCI tries to determine the user’s present psychological response. Third, an adaptive performance stimulus (APS) is triggered to enhance AE in opera performance, as determined by a rule base. To give the opera audience a high-quality viewing experience, the immersive theatre performance has been simulated. Fifty individuals have been used in the experimental assessment and performance studies. The findings demonstrated that the proposed technology had been able to accurately identify the decline in AE and that performing stimuli had a good impact on enhancing AE during an opera performance. It has been further shown that the suggested design improves the overall performance of AE by 5.8% when compared to a typical BCI design (one that uses EEG characteristics solely) for the proposed MESDT framework with BCI.

Imaginaries of the Pandemic in Chile: A Conceptual-Empirical Discussion.

This article aims to reconstruct the social imaginaries of Coronavirus disease 2019 (COVID‐19) in Chile. We seek to understand how families interpret their experience confronting the pandemic by identifying four main aspects: (a) the COVID‐19 pandemic, (b) working and learning, (c) health and (d) family life. Following Habermas' distinction between lifeworld and social systems, we consider these issues as constituting the social imaginary of lifeworld, different but related to the imaginaries of social systems. The qualitative empirical data was gathered through a sample of 38 families interviewed online between September 2020 and January 2021 in four Chilean cities: Iquique, Valparaíso, Santiago and Concepción. Other complementary sources of information are multimodal ethnography (digital diaries), press articles and state reports.

Review and Perspectives on Driver Digital Twin and Its Enabling Technologies for Intelligent Vehicles

Digital Twin (DT) is an emerging technology and has been introduced into intelligent driving and transportation systems to digitize and synergize connected automated vehicles. However, existing studies focus on the design of the automated vehicle, whereas the digitization of the human driver, who plays an important role in driving, is largely ignored. Furthermore, previous driver-related tasks are limited to specific scenarios and have limited applicability. Thus, a novel concept of a driver digital twin (DDT) is proposed in this study to bridge the gap between existing automated driving systems and fully digitized ones and aid in the development of a complete driving human cyber-physical system (H-CPS). This concept is essential for constructing a harmonious human-centric intelligent driving system that considers the proactivity and sensitivity of the human driver. The primary characteristics of the DDT include multimodal state fusion, personalized modeling, and time variance. Compared with the original DT, the proposed DDT emphasizes on internal personality and capability with respect to the external physiological-level state. This study systematically illustrates the DDT and outlines its key enabling aspects. The related technologies are comprehensively reviewed and discussed with a view to improving them by leveraging the DDT. In addition, the potential applications and unsettled challenges are considered. This study aims to provide fundamental theoretical support to researchers in determining the future scope of the DDT system

A federated learning method for real-time emotion state classification from multi-modal streaming.

Emotional and physical health are strongly connected and should be taken care of simultaneously to ensure completely healthy persons. A person's emotional health can be determined by detecting emotional states from various physiological measurements (EDA, RB, EEG, etc.). Affective Computing has become the field of interest, which uses software and hardware to detect emotional states. In the IoT era, wearable sensor-based real-time multi-modal emotion state classification has become one of the hottest topics. In such setting, a data stream is generated from wearable-sensor devices, data accessibility is restricted to those devices only and usually a high data generation rate should be processed to achieve real-time emotion state responses. Additionally, protecting the users' data privacy makes the processing of such data even more challenging. Traditional classifiers have limitations to achieve high accuracy of emotional state detection under demanding requirements of decentralized data and protecting users' privacy of sensitive information as such classifiers need to see all data. Here comes the federated learning, whose main idea is to create a global classifier without accessing the users' local data. Therefore, we have developed a federated learning framework for real-time emotion state classification using multi-modal physiological data streams from wearable sensors, called Fed-ReMECS. The main findings of our Fed-ReMECS framework are the development of an efficient and scalable real-time emotion classification system from distributed multimodal physiological data streams, where the global classifier is built without accessing (privacy protection) the users' data in an IoT environment. The experimental study is conducted using the popularly used multi-modal benchmark DEAP dataset for emotion classification. The results show the effectiveness of our developed approach in terms of accuracy, efficiency, scalability and users' data privacy protection.

Fractal of the Gobi surface sediment components and its variability characteristics

The surface sediment components of the Gobi are extremely complex, and the content has a “multimodal” distribution. Additionally, the gravel content, cover degree, and gravel distribution have shown significant spatial heterogeneity. Traditional fractal theory has limitations in the multimodal component fractal calculation, and the physical meaning remains unclear. In this study, we modified the traditional fractal calculation method according to a segmented weighted averaging method and quantitatively expressed spatial heterogeneity features, such as the Gobi surface, using the fractal dimension and its variation parameters. The results showed that the surface sediment component of Gobi was fractal, the fractal dimensions were between 2.23 and 2.84, and the average value was 2.56. Both the sediment component content and the granularity composition distribution structure (unimodal, bimodal, and multimodal state) had a significant influence on the fractal dimension values. The fractal dimensions of the medium-gravel Gobi sediment had a large local variability but less spatial variability. The fractal dimension degree of local variation of the sandy gravel Gobi sediment was low, but had a large range of spatial variation and a large spatial variability. The local variation and spatial variability of fine-gravel Gobi sediment were both small.

Multimodal perception links cellular state to decision-making in single cells.

Individual cells make decisions that are adapted to their internal state and surroundings, but how cells can reliably do this remains unclear. To study the information processing capacity of human cells, we conducted multiplexed quantification of signaling responses and markers of the cellular state. Signaling nodes in a network displayed adaptive information processing, which led to heterogeneous growth factor responses and enabled nodes to capture partially nonredundant information about the cellular state. Collectively, as a multimodal percept this gives individual cells a large information processing capacity to accurately place growth factor concentration within the context of their cellular state and make cellular state-dependent decisions. Heterogeneity and complexity in signaling networks may have coevolved to enable specific and context-aware cellular decision-making in a multicellular setting.

Alternative stable states and hydrological regime shifts in a large intermittent river

Non-perennial rivers and streams make up over half the global river network and are becoming more widespread. Transitions from perennial to non-perennial flow are a threshold-type change that can lead to alternative stable states in aquatic ecosystems, but it is unknown whether streamflow itself is stable in either wet (flowing) or dry (no-flow) conditions. Here, we investigated drivers and feedbacks associated with regime shifts between wet and dry conditions in an intermittent reach of the Arkansas River (USA) over the past 23 years. Multiple lines of evidence suggested that these regimes represent alternative stable states, including (a) significant jumps in discharge time series that were not accompanied by jumps in flow drivers such as precipitation and groundwater pumping; (b) a multi-modal state distribution with 92% of months experiencing no-flow conditions for <10% or >90% of days, despite unimodal distributions of precipitation and pumping; and (c) a hysteretic relationship between climate and flow state. Groundwater levels appear to be the primary control over the hydrological regime, as groundwater levels in the alluvial aquifer were higher than the stream stage during wet regimes and lower than the streambed during dry regimes. Groundwater level variation, in turn, was driven by processes occurring at both the regional scale (surface water inflows from upstream, groundwater pumping) and the reach scale (stream–aquifer exchange, diffuse recharge through the soil column). Historical regime shifts were associated with diverse pressures including network disconnection caused by upstream water use, increased flow stability potentially associated with reservoir operations, and anomalous wet and dry climate conditions. In sum, stabilizing feedbacks among upstream inflows, stream–aquifer interactions, climate, vegetation, and pumping appear to create alternative wet and dry stable states at this site. These stabilizing feedbacks suggest that widespread observed shifts from perennial to non-perennial flow will be difficult to reverse.

Multimodal driver state modeling through unsupervised learning

Naturalistic driving data (NDD) can help understand drivers’ reactions to each driving scenario and provide personalized context to driving behavior. However, NDD requires a high amount of manual labor to label certain driver’s state and behavioral patterns. Unsupervised analysis of NDD can be used to automatically detect different patterns from the driver and vehicle data. In this paper, we propose a methodology to understand changes in driver’s physiological responses within different driving patterns. Our methodology first decomposes a driving scenario by using a Bayesian Change Point detection model. We then apply the Latent Dirichlet Allocation method on both driver state and behavior data to detect patterns. We present two case studies in which vehicles were equipped to collect exterior, interior, and driver behavioral data. Four patterns of driving behaviors (i.e., harsh brake, normal brake, curved driving, and highway driving), as well as two patterns of driver’s heart rate (HR) (i.e., normal vs. abnormal high HR), and gaze entropy (i.e., low versus high), were detected in these two case studies. The findings of these case studies indicated that among our participants, the drivers’ HR had a higher fraction of abnormal patterns during harsh brakes, accelerating and curved driving. Additionally, free-flow driving with close to zero accelerations on the highway was accompanied by more fraction of normal HR as well as a lower gaze entropy pattern. With the proposed methodology we can better understand variations in driver’s psychophysiological states within different driving scenarios. The findings of this work, has the potential to guide future autonomous vehicles to take actions that are fit to each specific driver.

Multimodal EEG and Eye Tracking Feature Fusion Approaches for Attention Classification in Hybrid BCIs

Often, various modalities capture distinct aspects of particular mental states or activities. While machine learning algorithms can reliably predict numerous aspects of human cognition and behavior using a single modality, they can benefit from the combination of multiple modalities. This is why hybrid BCIs are gaining popularity. However, it is not always straightforward to combine features from a multimodal dataset. Along with the method for generating the features, one must decide when the modalities should be combined during the classification process. We compare unimodal EEG and eye tracking classification of internally and externally directed attention to multimodal approaches for early, middle, and late fusion in this study. On a binary dataset with a chance level of 0.5, late fusion of the data achieves the highest classification accuracy of 0.609–0.675 (95%-confidence interval). In general, the results indicate that for these modalities, middle or late fusion approaches are better suited than early fusion approaches. Additional validation of the observed trend will require the use of additional datasets, alternative feature generation mechanisms, decision rules, and neural network designs. We conclude with a set of premises that need to be considered when deciding on a multimodal attentional state classification approach.