Multimodal System Research Articles

Designing corporate mobility as a service – Decision support and perspectives

Corporate Mobility as a Service (CMaaS) integrates company-exclusive fleets and public mobility services, such as car sharing, bike sharing, or taxis, into one multimodal mobility system. As a result, companies may reduce their own vehicle fleet, their total mobility costs, as well as their emissions by shifting trips to alternative mobility modes. Against this background, we present a strategic-tactical decision support tool for corporate mobility managers to design a CMaaS system. We include strategic decisions on the fleet size and composition of company-exclusive fleets. The fleet composition addresses decisions on the inclusion of various mobility modes, e.g., cars and bikes, as well as on the choice of vehicle classes, e.g., regarding drive train technologies, and on how these vehicles are provided to the company, e.g., leased or owned vehicles. Further, we include tactical decisions on the choice of price tariffs offered by public mobility providers. Herein, our decision support tool accounts for structural requirements, such as the spatio-temporal corporate mobility demand, costs, upper bounds on the number of vehicles, technical vehicle restrictions, and the use of micromobility modes, like bikes and scooters. Finally, a comprehensive case study with ten mobility services and driving profiles of 144 companies enables us to draw general insights on the current and future potentials of CMaaS. Results promise average cost savings of 33% and a shift of 21% of trips to public mobility services.

Breath analysis system with convolutional neural network (CNN) for early detection of lung cancer

Early diagnosis of lung cancer, the leading cause of cancer-related death worldwide, is critical for reducing mortality rate. However, current diagnostic methods are invasive, time-consuming, costly, and may not always provide accurate diagnoses. For early diagnosis, recent research has focused on noninvasive approaches, including the detection of volatile organic compounds (VOCs) in human exhaled breath. Changes in the composition and concentration of VOCs in exhaled breath may indicate lung cancer, and this approach offers several advantages over traditional diagnostic methods. Moreover, the combination of a breath gas sensing system and machine learning algorithms provides a more accurate diagnosis. In this study, for the early diagnosis of lung cancer, a breath analysis system was developed using a gas sensor array and deep learning algorithm. The breath analysis system was designed to detect multiple VOCs in exhaled breath using ten semiconductor metal oxide (SMO), one photoionization detector (PID), nine electrochemical (EC) gas sensors. In total, 181 clinical breath samples (from 74 healthy controls and 107 lung cancer patients) were collected and analyzed using a 1D convolutional neural network (CNN) algorithm. The results showed an overall accuracy of 97.8% in classifying healthy controls and lung cancer patients using a complete clinical dataset. Through a comparison of the single-sensor type data and multimodal sensor data and performance analysis of three different deep learning models (multilayer perceptron, recurrent neural network, and CNN), we validated the potential of the breath analyzer with a multimodal sensor system and a 1D CNN as a lung cancer diagnostic device.

A multimodal emotion recognition system using deep convolution neural networks

Despite the progress in computer-technology, with regard to Human-Computer Interaction (HCI), emotion recognition is still a challenging problem. In this paper, we present a novel multimodal emotion recognition system capable of recognizing emotions from audio, video, and text data using deep convolution neural networks. The system is able to recognize happy, angry, sad, afraid, disgust, surprise and neutral emotions. We used three datasets to train and test the system, one set for each of the three input formats. The results show a recognition accuracy rate of 100% for audio, 69% for video, and 64% for text. When applying the decision-level fusion, the recorded accuracy rate is 80%. These results confirm that the system is effective in recognizing human emotions.

Deep learning based biometric authentication using electrocardiogram and iris

Authentication systems play an important role in wide range of applications. The traditional token certificate and password-based authentication systems are now replaced by biometric authentication systems. Generally, these authentication systems are based on the data obtained from face, iris, electrocardiogram (ECG), fingerprint and palm print. But these types of models are unimodal authentication, which suffer from accuracy and reliability issues. In this regard, multimodal biometric authentication systems have gained huge attention to develop the robust authentication systems. Moreover, the current development in deep learning schemes have proliferated to develop more robust architecture to overcome the issues of tradition machine learning based authentication systems. In this work, we have adopted ECG and iris data and trained the obtained features with the help of hybrid convolutional neural network- long short-term memory (CNN-LSTM) model. In ECG, R peak detection is considered as an important aspect for feature extraction and morphological features are extracted. Similarly, gabor-wavelet, gray level co-occurrence matrix (GLCM), gray level difference matrix (GLDM) and principal component analysis (PCA) based feature extraction methods are applied on iris data. The final feature vector is obtained from MIT-BIH and IIT Delhi Iris dataset which is trained and tested by using CNN-LSTM. The experimental analysis shows that the proposed approach achieves average accuracy, precision, and F1-core as 0.985, 0.962 and 0.975, respectively.

Redesigning large-scale multimodal transit networks with shared autonomous mobility services

This study addresses a large-scale multimodal transit network design problem, with Shared Autonomous Mobility Services (SAMS) as both transit feeders and an origin-to-destination mode. The framework captures spatial demand and modal characteristics, considers intermodal transfers and express services, determines transit infrastructure investment and path flows, and generates transit routes. A system-optimal multimodal transit network is designed with minimum total door-to-door generalized costs of users and operators, satisfying transit origin–destination demand within a pre-set infrastructure budget. Firstly, the geography, demand, and modes in each zone are characterized with continuous approximation. The decisions of network link investment and multimodal path flows in zonal connection optimization are formulated as a minimum-cost multi-commodity network flow (MCNF) problem and solved efficiently with a mixed-integer linear programming (MILP) solver. Subsequently, the route generation problem is solved by expanding the MCNF formulation to minimize intramodal transfers. The model is illustrated through a set of experiments with the Chicago network comprised of 50 zones and seven modes, under three scenarios. The computational results present savings in traveler journey time and operator cost demonstrating the potential benefits of collaboration between multimodal transit systems and SAMS.

Gas turbine aero engine fault detection using Geo-TLSVM and digital twin with multimodal data analysis

DT is the digital representation of physical systems to simulate real situations in a digital version. Here, the DT of the GaT-AE is created for the Fault Detection (FD) process. The GaT is a type of internal combustion engine that uses the air as the working fluid to propel the flight. However, the incorporation of big data analysis in the DT model is challenging in previous works. So, a multimodal GaT-AE fault identification system is proposed. First, the 3D print of the GaT-AE is created. Then, the data is collected from both built-in sensors and DT. Afterward, the multimodal data is pre-processed and balanced by using HT-QNN and QADASYN, respectively. Next, the features are processed by GT-PFS, and finally, the faults are detected by Geo-TLSVM. The analysis proved that the developed model outperformed the other state-of-the-art model with its effective FD rate of 97.24%.

Enhancement of sense of ownership using virtual and haptic feedback

Accomplishing motor function requires multimodal information, such as visual and haptic feedback, which induces a sense of ownership (SoO) over one’s own body part. In this study, we developed a visual–haptic human machine interface that combines three different types of feedback (visual, haptic, and kinesthetic) in the context of passive hand-grasping motion and aimed to generate SoO over a virtual hand. We tested two conditions, both conditions the three set of feedback were synchronous, the first condition was in-phase, and the second condition was in antiphase. In both conditions, we utilized passive visual feedback (pre-recorded video of a real hand displayed), haptic feedback (balloon inflated and deflated), and kinesthetic feedback (finger movement following the balloon curvature). To quantify the SoO, the participants’ reaction time was measured in response to a sense of threat. We found that most participants had a shorter reaction time under anti-phase condition, indicating that synchronous anti-phase of the multimodal system was better than in-phase condition for inducing a SoO of the virtual hand. We conclude that stronger haptic feedback has a key role in the SoO in accordance with visual information. Because the virtual hand is closing and the high pressure from the balloon against the hand creates the sensation of grasping and closing the hand, it appeared as though the person was closing his/her hand at the perceptual level.

An NLP-based system for modulating virtual experiences using speech instructions

The application of virtual reality (VR) technology in rehabilitation is a research area that has been attracting remarkable attention recently, especially as an intervention with learning and developmental disorders such as dyslexia and autism. Multi-modal VR-based rehabilitative systems can provide a rich experience to patients by responding to various types of bio-signals. This paper proposes a framework for incorporating voice input in rehabilitative games. The proposed framework equips collaborative learning environments with voice based interaction which allows for a more dynamic flow of therapy sessions between physically impaired patients and specialists in a rehabilitative VR environment. The proposed framework’s pipeline begins with a speech recognition module to parse utterances into sentences. Next, comes the feature extraction module to identify the intent of the user and entities as well. Finally, we modulate the 3D environment in Unity. The framework was designed in a way that guarantees smooth integration with any game or 3D environment. Several technologies were tested in terms of accuracy and speed to increase the overall performance of the pipeline. To the best of our knowledge, this research is the first to do the following: 1) build a dataset of instructions, inspired by a product already used in the healthcare market, 2) based on which we could realize a multi-modal system, with voice as one of the modalities, for the context of VR-based rehabilitation. We evaluate every stage in the system pipeline, and we show that we can achieve considerable results on each of these stages including the speech recognition module, with a 5.64% word error rate, and on intent classification where the BiLSTM model outperforms BERT in the Named Entity Recognition by 11% in terms of the F1-score. We envisage the proposed system to lend itself well to emerging healthcare frameworks in the Metaverse.

ОЦІНЮВАННЯ РІВНЯ ЕНЕРГОЕФЕКТИВНОСТІ АВІАЦІЙНИХ ПІДПРИЄМСТВ ТА АВІАЦІЙНОГО ТРАНСПОРТУ ДЛЯ ПІСЛЯВОЄННОГО ВІДНОВЛЕННЯ УКРАЇНИ

The article emphasizes that the question of researching the place of air transport in the aspect of multimodal transportation in the aspect of the post-war reconstruction of Ukraine is relevant. An analysis of the pre-war dynamics of the development of the transport industry of Ukraine is presented, which showed the insufficient level of development of the transport infrastructure and the organization of transport operations for both the transit country. Directions and approaches to the implementation of elements of a multimodal transport system using air transport are presented. It is proposed for further implementation on the territory of Ukraine, the schemes of multimodal transportation with the involvement of air transport, which are popular in the world practice. The proposed directions for the use of air transport in relation to cargo transportation in the aspect of multimodal transportation in the conditions of Ukraine have been determined. The study identifies the peculiarities of the integration of air transport into the transport system during the post-war reconstruction of Ukraine. For the development of aviation transport in the context of the development of transportation on the territory of Ukraine, it is necessary first of all to determine the strategic directions of the post-war development of the transport industry of Ukraine and potential resource opportunities for their solution. A method of increasing the energy efficiency and reliability of the power supply of aviation enterprises is proposed, which will allow performing a high-quality reconfiguration of the existing energy networks of aviation enterprises and performing a high-quality design of new modern infrastructure networks of the aviation industry.

Flexible Multimodal Sensing System Based on a Vertical Stacking Strategy for Efficiently Decoupling Multiple Signals.

Multisensory integration enables the simultaneous perception of multiple environmental stimuli while minimizing size and energy consumption. However, conventional multifunctional integration in flexible electronics typically requires large-scale horizontal sensing arrays (such as flexible printed circuit boards), posing decoupling complexities, tensile strain limitation, and spatial constraints. Herein, a fully flexible multimodal sensing system (FMSS) is developed by coupling biomimetic stretchable conductive films (BSCFs) and strain-insensitive communication interfaces using a vertical stacking integration strategy. The FMSS achieves vertical integration without additional adhesives, and it can incorporate individual sensing layers and stretchable interconnects without any essential constraint on their deformations. Accordingly, the temperature and pressure are precisely decoupled simultaneously, and tensile stress can be accurately discerned in different directions. This vertical stacking integration strategy is expected to offer a new approach to significantly streamline the design and fabrication of multimodal sensing systems and enhance their decoupling capabilities.

A low-cost automatic people-counting system at bus stops using Wi-Fi probe requests and deep learning

AbstractCounting people is an important part of people-centric applications, and the increase in the number of IoT devices has allowed the collection of huge amounts of data to facilitate people counting. The present study seeks to provide a novel, low-cost, automatic people-counting system for the use at bus stops, featuring a sniffing device that can capture Wi-Fi probe requests, and overcoming the problem of Media Access Control (MAC) randomization using deep learning. To make manual data collection considerably easier, a “People Counter” app was designed to collect ground truth data in order to train the model with higher accuracy. A user-friendly, operating system-independent dashboard was created to display the most relevant metrics. A two-step methodological approach was followed comprising device choice and data collection; data analysis and algorithm development. For the data analysis, three different approaches were tested, and among these a deep-learning approach using Convolutional Recurrent Neural Network (CRNN) with Long Short-term Memory (LSTM) architecture produced the best results. The optimal deep learning model predicted the number of people at the stop with a mean absolute error of ~ 1.2 persons, which can be considered a good preliminary result, considering that the experiment was done in a very complex open environment. People-counting systems at bus stops can support better bus scheduling, improve the boarding and alighting time of passengers, and aid the planning of integrated multi-modal transport system networks.

SecureSense: Enhancing Person Verification through Multimodal Biometrics for Robust Authentication

Biometrics provide enhanced security and convenience compared to conventional methods of individual authentication. A more robust and effective method of individual authentication has emerged due to recent advancements in multimodal biometrics. Unimodal systems offer lower security and lack the robustness found in multimodal biometric systems. The research paper introduces a novel approach, employing multiple biometric modalities, including face, fingerprint, and iris, to authenticate users in a multimodal biometric system. The paper proposes the ”Secure Sense” framework, which combines multiple biometric modalities to improve person verification accuracy. The proposed system utilizes both web-based and real-time datasets. For the web-based dataset, we employed the Chicago Face dataset for facial data, the MMU1 dataset for iris data, and the SOCOfing dataset for fingerprint data. In real-time data collection, facial data is captured using a Zebronics Zeb-Gem webcam, fingerprint data is obtained using the Mantra MFS scanner, and iris data is collected using the Mantra MIS scanner. In the envisioned system, we introduce an innovative approach that employs a decision-level fusion technique across three unique biometric modalities, resulting in an impressive accuracy rate of approximately 93% across all modalities.

NeuroIGN: Explainable Multimodal Image-Guided System for Precise Brain Tumor Surgery.

Precise neurosurgical guidance is critical for successful brain surgeries and plays a vital role in all phases of image-guided neurosurgery (IGN). Neuronavigation software enables real-time tracking of surgical tools, ensuring their presentation with high precision in relation to a virtual patient model. Therefore, this work focuses on the development of a novel multimodal IGN system, leveraging deep learning and explainable AI to enhance brain tumor surgery outcomes. The study establishes the clinical and technical requirements of the system for brain tumor surgeries. NeuroIGN adopts a modular architecture, including brain tumor segmentation, patient registration, and explainable output prediction, and integrates open-source packages into an interactive neuronavigational display. The NeuroIGN system components underwent validation and evaluation in both laboratory and simulated operating room (OR) settings. Experimental results demonstrated its accuracy in tumor segmentation and the success of ExplainAI in increasing the trust of medical professionals in deep learning. The proposed system was successfully assembled and set up within 11min in a pre-clinical OR setting with a tracking accuracy of 0.5 (± 0.1) mm. NeuroIGN was also evaluated as highly useful, with a high frame rate (19 FPS) and real-time ultrasound imaging capabilities. In conclusion, this paper describes not only the development of an open-source multimodal IGN system but also demonstrates the innovative application of deep learning and explainable AI algorithms in enhancing neuronavigation for brain tumor surgeries. By seamlessly integrating pre- and intra-operative patient image data with cutting-edge interventional devices, our experiments underscore the potential for deep learning models to improve the surgical treatment of brain tumors and long-term post-operative outcomes.

Perceptions of vulnerable roadway users on autonomous vehicle regulations

Introduction: Development and implementation of autonomous vehicle (AV) related regulations are necessary to ensure safe AV deployment and wide acceptance among all roadway users. Assessment of vulnerable roadway users’ perceptions on AV regulations could inform policymakers the development of appropriate AV regulations that facilitate the safety of diverse users in a multimodal transportation system. Method: This research evaluated pedestrians’ and bicyclists’ perceptions on six AV regulations (i.e., capping AV speed limit, operating AV in manual mode in the sensitive areas, having both pilot and co-pilot while operating AVs, and three data-sharing regulations). In addition, pedestrians’ and bicyclists’ perceptions of testing AVs in public streets were evaluated. Statistical testing and modeling techniques were applied to accomplish the research objectives. Results: Compared to the other AV regulations assessed in this research, strong support for AV-related data sharing regulations was identified. Older respondents showed higher approval of AV testing on public roadways and less support for regulating AVs. AV technology familiarity and safe road sharing perceptions with AVs resulted in lower support for AV regulations. Conclusions: Policymakers and AV technology developers could develop effective educational tools/resources to inform pedestrians and bicyclists about AV technology reliability and soften their stance, especially on AV regulations, which could delay technology development. Practical Applications: The findings of this research could be used to develop informed AV regulations and develop policies that could improve pedestrians’ and bicyclists’ attitudes/perceptions on regulating AVs and promoting AV technology deployments.

Humidity-mediated synaptic plasticity in Ag loaded porous SiOx based memristor for multimodal neuromorphic sensory system

Neuromorphic sensory system plays a critical role for human being to perceive, interact and even deduce with the external environment. Multimodal plasticity implementation of neuromorphic sensory system that can learn with diversified information empowers the development of environment-interactive artificial intelligence. In this work, we demonstrated a multimodal neuromorphic sensory system based on Ag loaded porous SiOx based memristor. The humidity-mediated synaptic plasticity behaviors were detailedly analyzed in the range of 10–90% relative humidity (RH). The humidity-mediated silver ion migration in porous SiOx memristors was studied by theoretical and experimental methods, and the mechanism of synergistic effect between porous micro-structure and ambient humidity was elucidated. A multimodal neuromorphic sensory system was finally constructed and the adaptive behavior of the human eye was also successfully simulated by taking advantage of this well-designed Au/Ag-SiOx/ITO memristor. The biomimetic intelligence demonstrated in our multimodal neuromorphic sensory devices and systems shows its potential in promoting the advancement in brain-like artificial intelligence.

Robotics multi-modal recognition system via computer-based vision

Robotics multi-modal recognition system via computer-based vision

A multimodal dialogue system for customer service based on user personality adaptation and dialogue strategies

ABSTRACT This paper presents a multimodal dialogue system with user personality adaptation and multiple nonverbal behavior generation for a conversational android robot that plays a customer service role in recommending travel plans. It is useful to estimate the user's personality to realize a dialogue system that adapts the dialogue strategy to individual users. To improve the user's impression of the conversation robot as a customer service agent, we must control the nonverbal behavior of the robot, such as its facial expressions and motions. Moreover, we need to design the dialogue strategy to provide information and make users enjoy the conversation by adding an ice-breaking step. Against this background, we develop a dialogue system prototype that adjusts the dialogue strategy based on the user's personality, as estimated by a pre-trained multimodal sensing model. Furthermore, we implement appropriate nonverbal behavior patterns for each situation, including the voice, motion, and facial expressions of the android robot, to provide it appropriate and polite service agent behaviors. A personality assessment game is introduced into the dialogue to prevent the user from becoming bored. The game plays an ice-breaking role in the dialogue. To let the user engage with the robot, we implement a complement to the user as a dialogue act. We evaluate the proposed multimodal dialogue system for customer service by means of participation in a dialogue robot competition. The fair evaluation results from 26 dialogue users who conversed with multiple systems show that the proposed dialogue system achieved the best score among 13 teams participating in the preliminary round of the competition, indicating that the user personality adaptation and various elements implemented in our dialogue system improve the dialogue experience of general users. The results of a third-party evaluation and multiple regression analysis show that natural response is the most important factor contributing to user impression.

Frame-based versus robot-assisted stereo-electro-encephalography for drug-resistant epilepsy.

Stereoelectroencephalography (SEEG) is an effective presurgical invasive evaluation for drug-resistant epilepsies. The introduction of robotic devices provides a simplified, accurate, and safe alternative to the conventional SEEG technique. We report our institutional experience with robot-assisted SEEG and compare its in vivo accuracy, operation efficiency, and safety with the more traditional SEEG workflow. All patients with medically refractory focal epilepsy who underwent SEEG depth electrode implantation between 2014 and 2022 were included in this study. Technical advancements of the robot-assisted technique are described. Analyses of patient demographics, electrode implantation accuracy, operation time, and procedure-related complications were performed. One hundred and sixty-six patients underwent 167 SEEG procedures. The first 141 procedures were performed using a conventional approach involving a Leksell stereotactic system, and the last 26 procedures were robot-assisted. Among the 1726 depth electrodes that were inserted, the median entry point localization error was as follows: conventional (1.0mm; range, 0.1-33.5mm) and robot-assisted (1.1mm; range, 0-4.8mm) (P = 0.17). The median target point localization error was as follows: conventional (2.8mm; range, 0.1-49mm) and robot-assisted (1.8mm; range, 0-30.3mm) (P < 0.001). The median operation time was significantly reduced with the robot-assisted workflow (90min vs. 77.5min; P < 0.01). Total complication rates were as follows: conventional (17.7%) and robot-assisted (11.5%) (P = 0.57). Major complication rates were 3.5% and 7.7% (P = 0.77), respectively. SEEG is a safe and highly accurate method that provides essential guidance for epilepsy surgery. Implementing SEEG in conjunction with multimodal planning systems and robotic devices can further increase safety margin, surgical efficiency, and accuracy.

Account based ticketing (ABT) integration for seamless multi-modal transport systems in smart cities

The digital transformation of urban transportation systems remains a critical challenge in Smart Cities' development. This paper addresses the pervasive issue of inadequate digitalization within public transportation systems. The paper highlights the promise of Account Based Ticketing (ABT) as a pivotal solution for integrating diverse public and private transportation modes allowing users to access different transportation services with a single account, without the need for new physical tickets or cards. ABT approach represents a breakthrough in seamlessly amalgamating various transportation operators' systems, transcending city limits into broader metropolitan spheres, and fostering integration and interoperability across transportation modes operated by disparate entities. Emphasizing beyond-city-scale integration, it explores the potential value of ABT to connect transportation networks extending into metropolitan influences. The proposed approach involves analyzing the implementation and functionality of ABT systems, underlining their capacity to overcome existing barriers in multi-modal transportation. The results anticipate enhanced accessibility, increased ridership, and streamlined user experiences, contributing to the realization of efficient and interconnected urban transport networks. This study not only underscores the value of ABT in revolutionizing contemporary transportation systems but also examines its broader implications for future urban planning and smart city development, highlighting the transformative potential and essential role that ABT plays in shaping the future of integrated transportation and Mobility as a Service in Smart Cities.

Enhanced multimodal biometric recognition systems based on deep learning and traditional methods in smart environments.

In the field of data security, biometric security is a significant emerging concern. The multimodal biometrics system with enhanced accuracy and detection rate for smart environments is still a significant challenge. The fusion of an electrocardiogram (ECG) signal with a fingerprint is an effective multimodal recognition system. In this work, unimodal and multimodal biometric systems using Convolutional Neural Network (CNN) are conducted and compared with traditional methods using different levels of fusion of fingerprint and ECG signal. This study is concerned with the evaluation of the effectiveness of proposed parallel and sequential multimodal biometric systems with various feature extraction and classification methods. Additionally, the performance of unimodal biometrics of ECG and fingerprint utilizing deep learning and traditional classification technique is examined. The suggested biometric systems were evaluated utilizing ECG (MIT-BIH) and fingerprint (FVC2004) databases. Additional tests are conducted to examine the suggested models with:1) virtual dataset without augmentation (ODB) and 2) virtual dataset with augmentation (VDB). The findings show that the optimum performance of the parallel multimodal achieved 0.96 Area Under the ROC Curve (AUC) and sequential multimodal achieved 0.99 AUC, in comparison to unimodal biometrics which achieved 0.87 and 0.99 AUCs, for the fingerprint and ECG biometrics, respectively. The overall performance of the proposed multimodal biometrics outperformed unimodal biometrics using CNN. Moreover, the performance of the suggested CNN model for ECG signal and sequential multimodal system based on neural network outperformed other systems. Lastly, the performance of the proposed systems is compared with previously existing works.