Multimodal Conditions Research Articles

Universal Fingerprint Generation: Controllable Diffusion Model with Multimodal Conditions.

The utilization of synthetic data for fingerprint recognition has garnered increased attention due to its potential to alleviate privacy concerns surrounding sensitive biometric data. However, current methods for generating fingerprints have limitations in creating impressions of the same finger with useful intra-class variations. To tackle this challenge, we present GenPrint, a framework to produce fingerprint images of various types while maintaining identity and offering humanly understandable control over different appearance factors such as fingerprint class, acquisition type, sensor device, and quality level. Unlike previous fingerprint generation approaches, GenPrint is not confined to replicating style characteristics from the training dataset alone: it enables the generation of novel styles from unseen devices without requiring additional fine-tuning. To accomplish these objectives, we developed GenPrint using latent diffusion models with multimodal conditions (text and image) for consistent generation of style and identity. Our experiments leverage a variety of publicly available datasets for training and evaluation. Results demonstrate the benefits of GenPrint in terms of identity preservation, explainable control, and universality of generated images. Importantly, the GenPrint-generated images yield comparable or even superior accuracy to models trained solely on real data and further enhances performance when augmenting the diversity of existing real fingerprint datasets.

Q residual non-parametric Distribution on Fault Detection Approach Using Unsupervised LSTM-KDE

It is well known among practitioner, majority collected data from industrial process plant are unlabeled. The collected historical data if utilize, able to provide vital information of process plant condition. Learning from unlabeled dataset, this study proposed Unsupervised LSTM-KDE approach as a measure to predict fault in industrial process plant. The residual based fault detection approach framework is utilized with long short-term memory (LSTM) as the main pattern learner for nonlinear and multimode condition that usually appear in process plant. Furthermore, kernel density approach (KDE) is used to determine the threshold value in non-parametric condition of unlabeled data. The LSTM-KDE approach later is evaluated with numerical data as well as Tennessee Eastman process plant dataset. The performance also was compared to Principal Component Analysis (PCA), Local outlier factor (LOF) and Auto-associative Kernel Regression (AAKR) to further examine the LSTM-KDE performance. The experimental results indicate that the LSTM-KDE fault detection approach has better learning performance and accuracy compared to other approaches

Incidental learning of collocations through different multimodal input: The role of learners’ initial L2 proficiency

This study investigated the impact of one unimodal (i.e., reading-only) and two different multimodal input modes (i.e., reading-while-listening and viewing with captions) on incidental collocation learning, alongside examining the relationship between learning gains and overall L2 proficiency. A total of 180 university students at pre-intermediate and upper-intermediate levels from China were randomly assigned to one of the input modes. Target collocation knowledge tests, including form recognition and meaning recall, were administered immediately after exposure to the input material and after a 2-week interval. Results indicated that while learners across different proficiency levels acquired collocation knowledge through all input modes to varying degrees, those in the multimodal conditions outperformed their counterparts in terms of both recognition and recall dimensions. Additionally, after 2 weeks, despite decreased learning gains for all learner groups, multimodal input was found to benefit upper-intermediate learners' meaning recall of the target collocations. Learners’ language proficiency significantly predicted their immediate learning outcomes in both recognition and recall. These findings offer further support for multimedia learning theory and dual code theory, contributing to a deeper understanding of the value of multimodal meaning-focused language input.

SketchDream: Sketch-based Text-To-3D Generation and Editing

Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.

Research Progress of EEG-Based Emotion Recognition: A Survey

Emotion recognition based on electroencephalography (EEG) signals has emerged as a prominent research field, facilitating objective evaluation of diseases like depression and motion detection for heathy people. Starting from the basic concepts of temporal-frequency-spatial features in EEG and the methods for cross-domain feature fusion, this survey then extends the overfitting challenge of EEG single-modal to the problem of heterogeneous modality modeling in multimodal conditions. It explores issues such as feature selection, sample scarcity, cross-subject emotional transfer, physiological knowledge discovery, multimodal fusion methods, and modality missing. These findings provide clues for researchers to further investigate emotion recognition based on EEG signals.

Fault Prognosis and Detection for Carbonate Chemical Process under Multimodal Conditions Based on Transformer and Self-Adaptive Deep Learning

Fault Prognosis and Detection for Carbonate Chemical Process under Multimodal Conditions Based on Transformer and Self-Adaptive Deep Learning

Developing a teacher agent to explore the role of gaze direction on young school children’s comprehension

Teachers’ gaze and school children’s language proficiency play a role for listening comprehension in the classroom. The influence of multimodal conditions on listening comprehension may be hard to study in a systematic manner with real speakers. The aim was to test the concept and feasibility of studying the influence of a speakers on passage comprehension performance in children using a digitally animated character. Twenty-four primary-school children were tested. A narrative passage was presented to the participants by a teacher agent. Half of the participants were randomized to a direct gaze condition and half to an averted gaze condition. The method is feasible with the intended population. We detected no differences in listening comprehension of the teacher agent’s gaze behavior in our sample. The concept works and it is feasible to test the effect of speaker gaze on children using the proposed methodology with some further development.

DAT: A robust Discriminant Analysis-based Test of unimodality for unknown input distributions

This letter introduces a Discriminant Analysis-based unimodality Test (DAT) for automatically identifying whether a dataset is unimodal or multimodal, detecting deviations in time series datasets, estimating statistical parameters, and identifying the skewness of the data. DAT is effective in classifying datasets under both unimodal and multimodal conditions and is suitable for bi-classification applications. The performance of DAT was compared to two well-known unimodality tests, namely the dip and folding tests, and is shown to perform better. We then extended DAT to the development of a fault detection technique, which was tested against five different machine learning classifiers using data from the Case Western Reserve University (CWRU) ball bearing database. The results obtained show that the proposed approach is effective, achieving 99.999% accuracy for detecting small ball bearing fault sizes (0.007 inches). Our conclusion indicates a significant potential of the proposed test for improving anomaly detection in industrial and other related fields.

On the spectral wave climate of the French Atlantic Ocean

The accurate description of wave climate at different spatio-temporal scales requires the application of advanced statistical models together with a good knowledge of geophysical processes. Furthermore, accurate modelling of multimodal sea states is of primary importance for most of offshore and coastal activities, such as wave energy device optimization, maritime design practice and for safety at sea. The present manuscript is conceived within this scientific framework proposing to assess wave climate of a complex area of French Atlantic Ocean bordering seas. The main “Spectral Climatology Types” of the area are identified as resulting of defined combinations of wave systems detected through directional spectra and partitioned wave systems analysis. The presence of swell systems is evaluated quantitatively on the whole area, characterized by different regimes of multimodal sea-states at strong regional variability and put into relation with main meteorological forcing active on the area at different spatial scales. Celtic Sea exhibits a marked regional characterization with a prevalence of multimodal conditions given by different combination of wave systems with increasing contribution of swell systems moving from North-West to South. A significant presence of crossing sea states is observed in South Celtic Sea especially in proximity of bathymetric slope. The South Bay of Biscay is influenced by fully developed swell systems enhanced by refraction effects caused by both the coastline and bathymetry gradients. English Channel and North Sea show complex sea-states conditions induced by local topography features together with wind channelling and tide effects able to trigger geophysical processes at a sub-scale responsible for the development of multimodal seas. Crossing and bimodal seas are also found to be influenced by bathymetry gradients acting directly on the directional spectrum shape as well as by tide due to tidal current-induced refraction effect on wave propagation. No generalized significant trends are detected within the investigated spatial domain for the wave spectrum integral quantities; locations sited at northern Celtic Sea show a downward Significant Wave Height, while only locations confined at eastern English Chanel only shows a Peak Period upward trend.

Neural correlates of individual differences in multimodal emotion recognition ability

Studies have reported substantial variability in emotion recognition ability (ERA) – an important social skill – but possible neural underpinnings for such individual differences are not well understood. This functional magnetic resonance imaging (fMRI) study investigated neural responses during emotion recognition in young adults (N = 49) who were selected for inclusion based on their performance (high or low) during previous testing of ERA. Participants were asked to judge brief video recordings in a forced-choice emotion recognition task, wherein stimuli were presented in visual, auditory and multimodal (audiovisual) blocks. Emotion recognition rates during brain scanning confirmed that individuals with high (vs low) ERA received higher accuracy for all presentation blocks. fMRI-analyses focused on key regions of interest (ROIs) involved in the processing of multimodal emotion expressions, based on previous meta-analyses. In neural response to emotional stimuli contrasted with neutral stimuli, individuals with high (vs low) ERA showed higher activation in the following ROIs during the multimodal condition: right middle superior temporal gyrus (mSTG), right posterior superior temporal sulcus (PSTS), and right inferior frontal cortex (IFC). Overall, results suggest that individual variability in ERA may be reflected across several stages of decisional processing, including extraction (mSTG), integration (PSTS) and evaluation (IFC) of emotional information.

Multi-modal Representation of the Size of Space in the Human Brain.

To estimate the size of an indoor space, we must analyze the visual boundaries that limit the spatial extent and acoustic cues from reflected interior surfaces. We used fMRI to examine how the brain processes the geometric size of indoor scenes when various types of sensory cues are presented individually or together. Specifically, we asked whether the size of space is represented in a modality-specific way or in an integrative way that combines multimodal cues. In a block-design study, images or sounds that depict small- and large-sized indoor spaces were presented. Visual stimuli were real-world pictures of empty spaces that were small or large. Auditory stimuli were sounds convolved with different reverberations. By using a multivoxel pattern classifier, we asked whether the two sizes of space can be classified in visual, auditory, and visual-auditory combined conditions. We identified both sensory-specific and multimodal representations of the size of space. To further investigate the nature of the multimodal region, we specifically examined whether it contained multimodal information in a coexistent or integrated form. We found that angular gyrus and the right medial frontal gyrus had modality-integrated representation, displaying sensitivity to the match in the spatial size information conveyed through image and sound. Background functional connectivity analysis further demonstrated that the connection between sensory-specific regions and modality-integrated regions increases in the multimodal condition compared with single modality conditions. Our results suggest that spatial size perception relies on both sensory-specific and multimodal representations, as well as their interplay during multimodal perception.

Implicit versus explicit processing of visual, olfactory, and multimodal landmark information in human wayfinding.

Despite the predominant focus on visual perception in most studies, the role of humans' sense of smell in navigation has often been neglected. Recent research, however, could show that humans are indeed able to use their sense of smell for orientation, particularly when processed implicitly. In this study, we investigate whether implicit perception of olfactory landmarks enhanced wayfinding performance compared to explicit perception. Fifty-two people completed a wayfinding and a recognition task in a virtual maze at two times of testing 1 month apart. Participants either received olfactory, visual, or both cues at the intersections. Wayfinding performance was better for olfactory landmarks, which were not correctly remembered in the recognition task. In contrast, wayfinding performance was better when visual landmarks were correctly remembered. In the multimodal condition, wayfinding performance was better with landmarks being remembered at t1 and remained the same at t2. Our results suggest distinct implicit processing mechanisms within the olfactory system and therefore hold important implications for the nature of spatial odor processing extending beyond explicit odor localization tasks. The study highlights the importance for future studies to develop and employ further experimental methods that capture implicit processing across all of our senses. This is crucial for a comprehensive understanding of consciousness, as olfaction strongly influences our behavior, but remains largely latent unless deliberately honed through practice.

A direct comparison of sound and vibration as sources of stimulation for a sensory substitution glove

Sensory substitution devices (SSDs) facilitate the detection of environmental information through enhancement of touch and/or hearing capabilities. Research has demonstrated that several tasks can be successfully completed using acoustic, vibrotactile, and multimodal devices. The suitability of a substituting modality is also mediated by the type of information required to perform the specific task. The present study tested the adequacy of touch and hearing in a grasping task by utilizing a sensory substitution glove. The substituting modalities inform, through increases in stimulation intensity, about the distance between the fingers and the objects. A psychophysical experiment of magnitude estimation was conducted. Forty blindfolded sighted participants discriminated equivalently the intensity of both vibrotactile and acoustic stimulation, although they experienced some difficulty with the more intense stimuli. Additionally, a grasping task involving cylindrical objects of varying diameters, distances and orientations was performed. Thirty blindfolded sighted participants were divided into vibration, sound, or multimodal groups. High performance was achieved (84% correct grasps) with equivalent success rate between groups. Movement variables showed more precision and confidence in the multimodal condition. Through a questionnaire, the multimodal group indicated their preference for using a multimodal SSD in daily life and identified vibration as their primary source of stimulation. These results demonstrate that there is an improvement in performance with specific-purpose SSDs, when the necessary information for a task is identified and coupled with the delivered stimulation. Furthermore, the results suggest that it is possible to achieve functional equivalence between substituting modalities when these previous steps are met.

Deep Learning-Based RGB-D Fusion for Multimodal Condition Assessment of Civil Infrastructure

Deep Learning-Based RGB-D Fusion for Multimodal Condition Assessment of Civil Infrastructure

SkateboardAI: The Coolest Video Action Recognition for Skateboarding (Student Abstract)

Impressed by the coolest skateboarding sports program from 2021 Tokyo Olympic Games, we are the first to curate the original real-world video datasets "SkateboardAI" in the wild, even self-design and implement diverse uni-modal and multi-modal video action recognition approaches to recognize different tricks accurately. For uni-modal methods, we separately apply (1)CNN and LSTM; (2)CNN and BiLSTM; (3)CNN and BiLSTM with effective attention mechanisms; (4)Transformer-based action recognition pipeline. Transferred to the multi-modal conditions, we investigated the two-stream Inflated-3D architecture on "SkateboardAI" datasets to compare its performance with uni-modal cases. In sum, our objective is developing an excellent AI sport referee for the coolest skateboarding competitions.

Assessing the Value of Multimodal Interfaces: A Study on Human-Machine Interaction in Weld Inspection Workstations.

Multimodal user interfaces promise natural and intuitive human-machine interactions. However, is the extra effort for the development of a complex multisensor system justified, or can users also be satisfied with only one input modality? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and the use of multiple input modalities is valuable and that it is difficult to predict the usability of individual input modalities for complex systems.

Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection

Cancer is a highly heterogeneous condition that can occur almost anywhere in the human body. [<sup>18</sup>F]fluorodeoxyglucose Positron Emission Tomography (<sup>18</sup>F-FDG PET) is a imaging modality commonly used to detect cancer due to its high sensitivity and clear visualisation of the pattern of metabolic activity. Nonetheless, as cancer is highly heterogeneous, it is challenging to train general-purpose discriminative cancer detection models, with data availability and disease complexity often cited as a limiting factor. Unsupervised learning methods, more specifically anomaly detection models, have been suggested as a putative solution. These models learn a healthy representation of tissue and detect cancer by predicting deviations from the healthy norm, which requires models capable of accurately learning long-range interactions between organs, their imaging patterns, and other abstract features with high levels of expressivity. Such characteristics are suitably satisfied by transformers, which have been shown to generate state-of-the-art results in unsupervised anomaly detection by training on normal data. This work expands upon such approaches by introducing multi-modal conditioning of the transformer via cross-attention i.e. supplying anatomical reference information from paired CT images to aid the PET anomaly detection task. Furthermore, we show the importance and impact of codebook sizing within a Vector Quantized Variational Autoencoder, on the ability of the transformer network to fulfill the task of anomaly detection. Using 294 whole-body PET/CT samples containing various cancer types, we show that our anomaly detection method is robust and capable of achieving accurate cancer localization results even in cases where normal training data is unavailable. In addition, we show the efficacy of this approach on out-of-sample data showcasing the generalizability of this approach even with limited training data. Lastly, we propose to combine model uncertainty with a new kernel density estimation approach, and show that it provides clinically and statistically significant improvements in accuracy and robustness, when compared to the classic residual-based anomaly maps. Overall, a superior performance is demonstrated against leading state-of-the-art alternatives, drawing attention to the potential of these approaches.

An immersive multisensory virtual reality approach to the study of human-built environment interactions

Although human-built environment interaction has been studied extensively, the scope has been mostly limited to a single sensory domain instead of multimodal conditions. This knowledge gap results partially from the intrinsic complexity of multisensory experiments in the field of built environment.This paper develops a novel method for studying human-built environment interaction under multimodal (visual, auditory and thermal) conditions using virtual reality technology. It enables dynamic interaction between the subjects and the virtual environment while collecting a comprehensive array of data regarding the human-environment interaction. A proof-of-concept study was conducted with the developed method in a semi-outdoor environmental context.The method can simultaneously maximise both internal and ecological validity. Furthermore, it enables a holistic understanding of the psychological, behavioural and physiological processes of the human-environment interaction. It can be applied to further research questions involving multisensory experience in the built environment, bridging the gap between empirical evidence and design implementation.

Auditory, tactile, and multimodal noise reduce balance variability

Auditory and somatosensory white noise can stabilize standing balance. However, the differential effects of auditory and tactile noise stimulation on balance are unknown. Prior work on unimodal noise stimulation showed gains in balance with white noise through the auditory and tactile modalities separately. The current study aims to examine whether multimodal noise elicits similar responses to unimodal noise. We recorded the postural sway of healthy young adults who were presented with continuous white noise through the auditory or tactile modalities and through a combination of both (multimodal condition) using a wearable device. Our results replicate previous work that showed that auditory or tactile noise reduces sway variability with and without vision. Additionally, we show that multimodal noise also reduces the variability of sway. Analysis of different frequency bands of sway is typically used to separate open-loop exploratory (< 0.3 Hz) and feedback-driven (> 0.3 Hz) sway. We performed this analysis and showed that unimodal and multimodal white noise affected postural sway variability similarly in both timescales. These results support that the sensory noise effects on balance are robust across unimodal and multimodal conditions and can affect both mechanisms of sway represented in the frequency spectrum. In future work, the parameters of acoustic/tactile manipulation should be optimized for the most effective balance stabilization, and multimodal therapies should be explored for older adults with typical age-related balance instabilities.

The effect of modality onset asynchrony and processing time on the recognition of text-supplemented speech.

This study examined the effect of modality onset asynchrony and response processing time for the recognition of text-supplemented speech. Speech and text were periodically interrupted by noise or black bars, respectively, to preserve 50% of the sentence and presented in unimodal and multimodal conditions. Sentence recognition and response errors were assessed for responses made simultaneous with the stimulus or after its presentation. Increased processing time allowed for the cognitive repair of initial response errors in working memory. Text-supplemented speech was best recognized with minimal temporal asynchrony. Overall, text supplementation facilitated the recognition of degraded speech when provided sufficient processing time.