Multimodal Architecture Research Articles

Supervised Detection of Connector Lock Events with Optical Microphone Data.

In manufacturing industry, one of the main targets is to increase automation and ultimately to avoid failures under all circumstances. The plugging and locking of connectors is a class of tasks which is yet hard to be automatized with sufficiently high process stability. Due to the variation of plugging positions and external disturbances, e.g. occlusion due to cables, the quality assessment of plugging processes has emerged as a challenging task for image-based systems. For this reason, the proposed approach analyzes the inherent acoustic connector locking properties in combination with different neural network architectures in order to correctly identify connector locking signals and further to distinguish them from other machining events occurring in assembly plants. For this specific task, highly sensitive optical microphones have been applied for data acquisition. The proposed experiments are carried out under laboratory conditions as well as for the more complex situation in a real manufacturing environment. In this context, the usage of multimodal neural network architectures achieved highest levels in classification performance with accuracy levels close to 90%.

Random Interval Attendance Management System (RIAMS): A Novel Multimodal Approach for Post-COVID Virtual Learning

The exceptional circumstance caused by the COVID-19 pandemic demands substantial modifications in the teaching-learning processes across the globe. Teachers and students are making use of online learning in virtual classrooms as an alternative for face-to-face learning in physical classrooms. However, students’ attendance management during virtual learning is a challenging problem. It is quite difficult to identify students’ disengagement and even to know whether they are in front of their smart devices or not. In this paper, we introduce the ‘Random Interval Attendance Management System’ (RIAMS), which is an innovative solution for attendance monitoring issues, students’ disengagement, and attendance faking during virtual learning. In RIAMS, we employed a face recognition module built using the Dlib open-source software library. In order to improve the efficiency of the system, we introduced two ancillary modalities – verifying students’ responses to CAPTCHAs and UIN (Unique Identification Number) queries. Both the face recognition and ancillary modalities operate at random intervals of time. This distinctive feature of randomness in our design ensures that students’ attention and engagement in virtual learning are enhanced. Furthermore, the RIAMS’ multimodal architecture and its sub-modalities’ adaptive weight system enable teachers to customize their attendance strategy for every course. The output analysis of each of the RIAMS modalities and the combined results emphasize the effectiveness and reliability of our system in the attendance management for virtual learning. The novel RIAMS model has the potential to be extensively deployed for virtual learning in post-COVID settings.

Multi-mode Systems for Resilient Security in Industry 4.0

Abstract In the era of the Internet of Things and Industry 4.0, machines and devices are increasingly getting connected. These connections go hand in hand with security vulnerabilities and potential threats to these devices. In regular IT systems, we typically provide updates to eliminate vulnerabilities. In industrial automation and control systems, especially in mass production, legacy systems are widespread and installing updates causes downtime. Availability is one of the top goals; stopping a machine in case of a cyber-security threat is often too expensive. But, system integrators and asset owners should not have to wait until product or component suppliers release appropriate updates. Due to safety and warranty requirements, developing and distributing updates can take a long time. In the meantime, attackers can pose threats by taking advantage of devices’ known vulnerabilities. In this paper, we propose the design of resilient systems based on multi-modal architectures with several operational modes. When vulnerabilities of systems become known, or when systems get even attacked at some point, mode switching can overcome the time between vulnerability disclosure or attack, and the availability of corresponding security patches. Therefore, system integrators and asset owners can actively protect themselves by implicitly or explicitly switching to modes with reduced attack surfaces and, thus, with limited ranges of activity for attackers.

Using 3-D Convolution and Multimodal Architecture for Earthquake Damage Detection Based on Satellite Imagery and Digital Urban Data

When a large earthquake occurs, it is quite important to quickly figure out the damage distribution of housing structures for disaster prevention measures. Currently, the information is confirmed manually by local public organizations, which takes a lot of time. Therefore, a method is required for gathering the information more swiftly and objectively. In this work, a novel method for detecting damage to single buildings from a set of multitemporal satellite images is developed by applying a recent machine learning approach. The damage detection system is designed as a deep learning model that uses multimodal data, consisting of optical satellite images and structural attributes. The proposed method achieved over 90% detection accuracy on damaged housing in the affected area of 2016 Kumamoto earthquake, Japan from satellite images taken by Pleiades as well as digital urban data.

Capsule Attention for Multimodal EEG-EOG Representation Learning With Application to Driver Vigilance Estimation.

Driver vigilance estimation is an important task for transportation safety. Wearable and portable brain-computer interface devices provide a powerful means for real-time monitoring of the vigilance level of drivers to help with avoiding distracted or impaired driving. In this paper, we propose a novel multimodal architecture for in-vehicle vigilance estimation from Electroencephalogram and Electrooculogram. To enable the system to focus on the most salient parts of the learned multimodal representations, we propose an architecture composed of a capsule attention mechanism following a deep Long Short-Term Memory (LSTM) network. Our model learns hierarchical dependencies in the data through the LSTM and capsule feature representation layers. To better explore the discriminative ability of the learned representations, we study the effect of the proposed capsule attention mechanism including the number of dynamic routing iterations as well as other parameters. Experiments show the robustness of our method by outperforming other solutions and baseline techniques, setting a new state-of-the-art. We then provide an analysis on different frequency bands and brain regions to evaluate their suitability for driver vigilance estimation. Lastly, an analysis on the role of capsule attention, multimodality, and robustness to noise is performed, highlighting the advantages of our approach.

Multimodal biometric recognition systems using deep learning based on the finger vein and finger knuckle print fusion

Recognition systems using multimodal biometrics attracts attention because they improve recognition efficiency and high-security level compared to the unimodal biometrics system. In this study, the authors present a secure multimodal biometrics recognition system based on the deep learning method that uses convolutional neural networks (CNNs). The authors propose two multimodal architectures using the finger knuckle print (FKP) and the finger vein (FV) biometrics with different levels of fusion: the features level fusion and scores level fusion. The features extraction for FKP and FV are performed using transfer learning CNN architectures: AlexNet, VGG16, and ResNet50. The key step aims to select separate features descriptors from each unimodal biometrics modality. After that, the authors combine them using the proposed fusion approaches were support vector machine or Softmax applies as classifiers to increase the proposed system security. The efficiency of the proposed algorithms is tested using publicly available biometrics databases. The experimental results show that the proposed fusion architectures achieve an accuracy of 99.89% and an equal error rate of 0.05%. The obtained results indicate that the biometrics recognition system using deep learning is secure, robust, and reliable.

SAIF: A Correction-Detection Deep-Learning Architecture for Personal Assistants.

Intelligent agents that can interact with users using natural language are becoming increasingly common. Sometimes an intelligent agent may not correctly understand a user command or may not perform it properly. In such cases, the user might try a second time by giving the agent another, slightly different command. Giving an agent the ability to detect such user corrections might help it fix its own mistakes and avoid making them in the future. In this work, we consider the problem of automatically detecting user corrections using deep learning. We develop a multimodal architecture called SAIF, which detects such user corrections, taking as inputs the user’s voice commands as well as their transcripts. Voice inputs allow SAIF to take advantage of sound cues, such as tone, speed, and word emphasis. In addition to sound cues, our model uses transcripts to determine whether a command is a correction to the previous command. Our model also obtains internal input from the agent, indicating whether the previous command was executed successfully or not. Finally, we release a unique dataset in which users interacted with an intelligent agent assistant, by giving it commands. This dataset includes labels on pairs of consecutive commands, which indicate whether the latter command is in fact a correction of the former command. We show that SAIF outperforms current state-of-the-art methods on this dataset.

Deep learning-based late fusion of multimodal information for emotion classification of music video

Affective computing is an emerging area of research that aims to enable intelligent systems to recognize, feel, infer and interpret human emotions. The widely spread online and off-line music videos are one of the rich sources of human emotion analysis because it integrates the composer’s internal feeling through song lyrics, musical instruments performance and visual expression. In general, the metadata which music video customers to choose a product includes high-level semantics like emotion so that automatic emotion analysis might be necessary. In this research area, however, the lack of a labeled dataset is a major problem. Therefore, we first construct a balanced music video emotion dataset including diversity of territory, language, culture and musical instruments. We test this dataset over four unimodal and four multimodal convolutional neural networks (CNN) of music and video. First, we separately fine-tuned each pre-trained unimodal CNN and test the performance on unseen data. In addition, we train a 1-dimensional CNN-based music emotion classifier with raw waveform input. The comparative analysis of each unimodal classifier over various optimizers is made to find the best model that can be integrate into a multimodal structure. The best unimodal modality is integrated with corresponding music and video network features for multimodal classifier. The multimodal structure integrates whole music video features and makes final classification with the SoftMax classifier by a late feature fusion strategy. All possible multimodal structures are also combined into one predictive model to get the overall prediction. All the proposed multimodal structure uses cross-validation to overcome the data scarcity problem (overfitting) at the decision level. The evaluation results using various metrics show a boost in the performance of the multimodal architectures compared to each unimodal emotion classifier. The predictive model by integration of all multimodal structure achieves 88.56% in accuracy, 0.88 in f1-score, and 0.987 in area under the curve (AUC) score. The result suggests human high-level emotions are automatically well classified in the proposed CNN-based multimodal networks, even though a small amount of labeled data samples is available for training.

A Multimodal Localization Framework Design for IoT Applications.

Multiple Radio Access Technology (multi-RAT) communication with Low Power Wide Area Networks (LPWAN) significantly increases the flexibility of Internet of Things (IoT) applications. Location-based services that build upon such a multimodal communication architecture are able to switch to an optimal localization method depending on the constraints of the active wireless technology. Furthermore, the resulting location estimate can aid location-based handover mechanisms to reduce the energy consumption of a multi-RAT IoT device. In this research, we present our design of a multimodal localization framework and illustrate the benefit of such a framework with two IoT use case examples. For the first use case, valuable artwork is tracked during transportation to a museum. In the second use case, we monitor the usage and location of large construction tools. Finally, we propose how our localization framework can be improved to deal with implementation challenges and to reduce location estimation errors.

Highly sensitive detection of dopamine based on hierarchical nanoporous NiCoO2/Ni composite

Abstract Hierarchical nanoporous (HNP) NiCoO2/Ni composite is directly fabricated by one-step dealloying of NiCoAl ternary alloy in mild alkaline solution. HNP-NiCoO2/Ni composite is composed of three dimensional interlinking ligament backbone and open hollow channels with rich mesopores and macropores penetrating through. HNP-NiCoO2/Ni possesses preferable structure advantages to construct sensing platform in term of large surface area, robust multimodal porous architecture, as well the incorporation of well conductive Ni. As is expected, exceptional electrochemical sensing performances are achieved over HNP-NiCoO2/Ni with high sensitivity, superior selectivity, and long term durability toward DA assay even without any activation pretreatment. Especially, HNP-NiCoO2/Ni exhibits a wide linear detection range from 1 to 800 μM with a rapid response in ~2.8 s and a detection limit as low as 0.12 μM. Furthermore, HNP-NiCoO2/Ni also shows a good anti-interference toward DA detection in the presence of AC, glucose, H2O2, Cl−, K+, and Na+. Along with the merits of simple preparation, low cost, and excellent electrocatalytic performances, the as-prepared HNP-NiCoO2/Ni presents promising application potential to construct highly sensitive electrochemical sensors for DA assay.

Grounded Sequence to Sequence Transduction

Speech recognition and machine translation have made major progress over the past decades, providing practical systems to map one language sequence to another. Although multiple modalities such as sound and video are becoming increasingly available, the state-of-the-art systems are inherently unimodal, in the sense that they take a single modality — either speech or text — as input. Evidence from human learning suggests that additional modalities can provide disambiguating signals crucial for many language tasks. In this article, we describe the How2 dataset , a large, open-domain collection of videos with transcriptions and their translations. We then show how this single dataset can be used to develop systems for a variety of language tasks and present a number of models meant as starting points. Across tasks, we find that building multimodal architectures that perform better than their unimodal counterpart remains a challenge. This leaves plenty of room for the exploration of more advanced solutions that fully exploit the multimodal nature of the How2 dataset , and the general direction of multimodal learning with other datasets as well.

Automatic Mobile App Identification From Encrypted Traffic With Hybrid Neural Networks

The proliferation of handheld devices has led to an explosive growth of mobile traffic volumes on the Internet. Identifying mobile apps from network traffic has become a crucial task for mobile network management and security. Traditionally, the design of accurate identifiers relies on the deep packet inspection (DPI) techniques. However, such approaches have become less effective with the raising adoption of encrypted protocols in mobile applications (mostly TLS). To address the problem, various machine learning methods have been studied and used. Most of them use linear classifiers on top of hand-engineered features, which are unreliable due to the complexity of mobile traffic. In this article we propose App-Net, an end-to-end hybrid neural network for mobile app identification from encrypted TLS traffic. App-Net is designed by combining RNN and CNN in a parallel way and can automatically learn effective features from raw TLS flows. With coordinated fusion and optimized training, the hybrid and multimodal architecture is able to characterize both flow sequence patterns and app signatures to learn a joint flow-app embedding. We evaluate App-Net on a real-world dataset covering 80 apps. The results show that our method can achieve an excellent performance and outperform the state-of-the-art methods.

Cyber Physical and Social Networks in IoV (CPSN-IoV): A Multimodal Architecture in Edge-Based Networks for Optimal Route Selection Using 5G Technologies

Humans are blessed with the intelligence to create links, develop semantic metaphors and models for reasoning; construct rules for decision making; and to form bounded loops for interaction, socialization and knowledge sharing. But machines are inadequate with these extraordinary abilities rather, numerous algorithms and mathematical models can be used to connect physical resources with cyberspaces to control objects and, develop cognitive learning for optimal decision making. Connected users and devices in closed virtual and physical proximity give direction towards the plethora of real-world applications for physical, social and, cyber computing. Because of the increase in social media networking and 5G communication links offer real-time crowdsourcing and sensing as a complementary base for information. Proceeding this idea, in this study we have proposed Cyber-Physical and Social Networks (CPSN) for two fundamental operations in IoV (Internet of Vehicles) as CPSN-IoV; (1) to define conceptual architecture of CPSN-IoV for data-oriented network for smart infrastructure and, (2) to create the significant virtual space where the instances of smart vehicles, devices, and things will have meaningful links with the real world objects where, CPSN-IoV will evolve, emerge, compete, and collaborate with all connected objects to strengthen the decision making process. To investigate the potential impact of our proposed study, we have simulated the taxicab trajectory data of the urban city of Portugal in OMNeT++ for the in-depth understanding of road topology, connected vehicles and things, and their traffic trends; and users’ social media streams in respective edge for efficient route planning. The results of simulation demonstrate that our proposed framework has the ability to achieve human-machine intellectual association for managing the smart environment.

Self-Supervised Model Adaptation for Multimodal Semantic Segmentation

Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the real-world. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering the model incapable of focusing only on relevant complementary information for fusion. To address this limitation, we propose a mutimodal semantic segmentation framework that dynamically adapts the fusion of modality-specific features while being sensitive to the object category, spatial location and scene context in a self-supervised manner. Specifically, we propose an architecture consisting of two modality-specific encoder streams that fuse intermediate encoder representations into a single decoder using our proposed self-supervised model adaptation fusion mechanism which optimally combines complementary features. As intermediate representations are not aligned across modalities, we introduce an attention scheme for better correlation. In addition, we propose a computationally efficient unimodal segmentation architecture termed AdapNet++ that incorporates a new encoder with multiscale residual units and an efficient atrous spatial pyramid pooling that has a larger effective receptive field with more than 10x fewer parameters, complemented with a strong decoder with a multi-resolution supervision scheme that recovers high-resolution details. Comprehensive empirical evaluations on several benchmarks demonstrate that both our unimodal and multimodal architectures achieve state-of-the-art performance.

An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks

Recently, there has been growing use of deep neural networks in many modern speech-based systems such as speaker recognition, speech enhancement, and emotion recognition. Inspired by this success, we propose to address the task of voice activity detection (VAD) by incorporating auditory and visual modalities into an end-to-end deep neural network. We evaluate our proposed system in challenging acoustic environments including high levels of noise and transients, which are common in real-life scenarios. Our multimodal setting includes a speech signal captured by a microphone and a corresponding video signal capturing the speaker's mouth region. Under such difficult conditions, robust features need to be extracted from both modalities in order for the system to accurately distinguish between speech and noise. For this purpose, we utilize a deep residual network, to extract features from the video signal, while for the audio modality, we employ a variant of WaveNet encoder for feature extraction. The features from both modalities are fused using multimodal compact bilinear pooling to form a joint representation of the speech signal. To further encode the temporal information, we feed the fused signal to a long short-term memory network and the system is then trained in an end-to-end supervised fashion. Experimental results demonstrate the improved performance of the proposed end-to-end multimodal architecture compared to unimodal variants for VAD. Upon the publication of this paper, we will make the implementation of our proposed models publicly available at https://github.com/iariav/End-to-End-VAD and https://israelcohen.com .

A Spam Filtering Method Based on Multi-Modal Fusion

In recent years, the single-modal spam filtering systems have had a high detection rate for image spamming or text spamming. To avoid detection based on the single-modal spam filtering systems, spammers inject junk information into the multi-modality part of an email and combine them to reduce the recognition rate of the single-modal spam filtering systems, thereby implementing the purpose of evading detection. In view of this situation, a new model called multi-modal architecture based on model fusion (MMA-MF) is proposed, which use a multi-modal fusion method to ensure it could effectively filter spam whether it is hidden in the text or in the image. The model fuses a Convolutional Neural Network (CNN) model and a Long Short-Term Memory (LSTM) model to filter spam. Using the LSTM model and the CNN model to process the text and image parts of an email separately to obtain two classification probability values, then the two classification probability values are incorporated into a fusion model to identify whether the email is spam or not. For the hyperparameters of the MMA-MF model, we use a grid search optimization method to get the most suitable hyperparameters for it, and employ a k-fold cross-validation method to evaluate the performance of this model. Our experimental results show that this model is superior to the traditional spam filtering systems and can achieve accuracies in the range of 92.64–98.48%.

Unified platform for multimodal voxel-based analysis to evaluate tumour perfusion and diffusion characteristics before and after radiation treatment evaluated in metastatic brain cancer.

Early changes in tumour behaviour following stereotactic radiosurgery) are potential biomarkers of response. To-date quantitative model-based measures of dynamic contrast-enhanced (DCE) and diffusion-weighted (DW) MRI parameters have shown widely variable findings, which may be attributable to variability in image acquisition, post-processing and analysis. Big data analytic approaches are needed for the automation of computationally intensive modelling calculations for every voxel, independent of observer interpretation. This unified platform is a voxel-based, multimodality architecture that brings complimentary solute transport processes such as perfusion and diffusion into a common framework. The methodology was tested on synthetic data and digital reference objects and consequently evaluated in patients who underwent volumetric DCE-CT, DCE-MRI and DWI-MRI scans before and after treatment. Three-dimensional pharmacokinetic parameter maps from both modalities were compared as well as the correlation between apparent diffusion coefficient (ADC) values and the extravascular, extracellular volume (Ve). Comparison of histogram parameters was done via Bland-Altman analysis, as well as Student's t-test and Pearson's correlation using two-sided analysis. System testing on synthetic Tofts model data and digital reference objects recovered the ground truth parameters with mean relative percent error of 1.07 × 10-7 and 5.60 × 10-4 respectively. Direct voxel-to-voxel Pearson's analysis showed statistically significant correlations between CT and MR which peaked at Day 7 for Ktrans (R = 0.74, p <= 0.0001). Statistically significant correlations were also present between ADC and Ve derived from both DCE-MRI and DCE-CT with highest median correlations found at Day 3 between median ADC and Ve,MRI values (R = 0.6, p < 0.01) The strongest correlation to DCE-CT measurements was found with DCE-MRI analysis using voxelwise T10 maps (R = 0.575, p < 0.001) instead of assigning a fixed T10 value. The unified implementation of multiparametric transport modelling allowed for more robust and timely observer-independent data analytics. Utility of a common analysis platform has shown higher correlations between pharmacokinetic parameters obtained from different modalities than has previously been reported. Utility of a common analysis platform has shown statistically higher correlations between pharmacokinetic parameters obtained from different modalities than has previously been reported.

Multimodal Generic Framework for Multimedia Documents Adaptation

Today, people are increasingly capable of creating and sharing documents (which generally are multimedia oriented) via the internet. These multimedia documents can be accessed at anytime and anywhere (city, home, etc.) on a wide variety of devices, such as laptops, tablets and smartphones. The heterogeneity of devices and user preferences has raised a serious issue for multimedia contents adaptation. Our research focuses on multimedia documents adaptation with a strong focus on interaction with users and exploration of multimodality. We propose a multimodal framework for adapting multimedia documents based on a distributed implementation of W3C’s Multimodal Architecture and Interfaces applied to ubiquitous computing. The core of our proposed architecture is the presence of a smart interaction manager that accepts context related information from sensors in the environment as well as from other sources, including information available on the web and multimodal user inputs. The interaction manager integrates and reasons over this information to predict the user’s situation and service use. A key to realizing this framework is the use of an ontology that undergirds the communication and representation, and the use of the cloud to insure the service continuity on heterogeneous mobile devices. Smart city is assumed as the reference scenario.

Towards a Generic Multimodal Architecture for Batch and Streaming Big Data Integration

Big Data are rapidly produced from various heterogeneous data sources. They are of different types (text, image, video or audio) and have different levels of reliability and completeness. One of the most interesting architectures that deal with the large amount of emerging data at high velocity is called the lambda architecture. In fact, it combines two different processing layers namely batch and speed layers, each providing specific views of data while ensuring robustness, fast and scalable data processing. However, most papers dealing with the lambda architecture are focusing one single type of data generally produced by a single data source. Besides, the layers of the architecture are implemented independently, or, at best, are combined to perform basic processing without assessing either the data reliability or completeness. Therefore, inspired by the lambda architecture, we propose in this paper a generic multimodal architecture that combines both batch and streaming processing in order to build a complete, global and accurate insight in near-real-time based on the knowledge extracted from multiple heterogeneous Big Data sources. Our architecture uses batch processing to analyze the data structures and contents, build the learning models and calculate the reliability index of the involved sources, while the streaming processing uses the built-in models of the batch layer to immediately process incoming data and rapidly provide results. We validate our architecture in the context of urban traffic management systems in order to detect congestions.

M-VAD names: a dataset for video captioning with naming

Current movie captioning architectures are not capable of mentioning characters with their proper name, replacing them with a generic "someone" tag. The lack of movie description datasets with characters' visual annotations surely plays a relevant role in this shortage. Recently, we proposed to extend the M-VAD dataset by introducing such information. In this paper, we present an improved version of the dataset, namely M-VAD Names, and its semi-automatic annotation procedure. The resulting dataset contains 63k visual tracks and 34k textual mentions, all associated with character identities. To showcase the features of the dataset and quantify the complexity of the naming task, we investigate multimodal architectures to replace the "someone" tags with proper character names in existing video captions. The evaluation is further extended by testing this application on videos outside of the M-VAD Names dataset.