Cross-modal Transfer Research Articles

Visuospatial attention revamps cortical processing of sound amid audiovisual uncertainty.

Selective attentional biases arising from one sensory modality manifest in others. The effects of visuospatial attention, important in visual object perception, are unclear in the auditory domain during audiovisual (AV) scene processing. We investigate temporal and spatial factors that underlie such transfer neurally. Auditory encoding of random tone pips in AV scenes was addressed via a temporal response function model (TRF) of participants' electroencephalogram (N = 30). The spatially uninformative pips were associated with spatially distributed visual contrast reversals ("flips"), through asynchronous probabilistic AV temporal onset distributions. Participants deployed visuospatial selection on these AV stimuli to perform a task. A late (~300 ms) cross-modal influence over the neural representation of pips was found in the original and a replication study (N = 21). Transfer depended on selected visual input being (i) presented during or shortly after a related sound, in relatively limited temporal distributions (<165 ms); (ii) positioned across limited (1:4) visual foreground to background ratios. Neural encoding of auditory input, as a function of visual input, was largest at visual foreground quadrant sectors and lowest at locations opposite to the target. The results indicate that ongoing neural representations of sounds incorporate visuospatial attributes for auditory stream segregation, as cross-modal transfer conveys information that specifies the identity of multisensory signals. A potential mechanism is by enhancing or recalibrating the tuning properties of the auditory populations that represent them as objects. The results account for the dynamic evolution under visual attention of multisensory integration, specifying critical latencies at which relevant cortical networks operate.

Multi-modal body part segmentation of infants using deep learning

BackgroundMonitoring the body temperature of premature infants is vital, as it allows optimal temperature control and may provide early warning signs for severe diseases such as sepsis. Thermography may be a non-contact and wireless alternative to state-of-the-art, cable-based methods. For monitoring use in clinical practice, automatic segmentation of the different body regions is necessary due to the movement of the infant.MethodsThis work presents and evaluates algorithms for automatic segmentation of infant body parts using deep learning methods. Based on a U-Net architecture, three neural networks were developed and compared. While the first two only used one imaging modality (visible light or thermography), the third applied a feature fusion of both. For training and evaluation, a dataset containing 600 visible light and 600 thermography images from 20 recordings of infants was created and manually labeled. In addition, we used transfer learning on publicly available datasets of adults in combination with data augmentation to improve the segmentation results.ResultsIndividual optimization of the three deep learning models revealed that transfer learning and data augmentation improved segmentation regardless of the imaging modality. The fusion model achieved the best results during the final evaluation with a mean Intersection-over-Union (mIoU) of 0.85, closely followed by the RGB model. Only the thermography model achieved a lower accuracy (mIoU of 0.75). The results of the individual classes showed that all body parts were well-segmented, only the accuracy on the torso is inferior since the models struggle when only small areas of the skin are visible.ConclusionThe presented multi-modal neural networks represent a new approach to the problem of infant body segmentation with limited available data. Robust results were obtained by applying feature fusion, cross-modality transfer learning and classical augmentation strategies.

The role of spatial information in an approximate cross-modal number matching task

The approximate number system (ANS) is thought to be an innate cognitive system that allows humans to perceive numbers (>4) in a fuzzy manner. One assumption of the ANS is that numerosity is represented amodally due to a mechanism, which filters out nonnumerical information from stimulus material. However, some studies show that nonnumerical information (e.g., spatial parameters) influence the numerosity percept as well. Here, we investigated whether there is a cross-modal transfer of spatial information between the haptic and visual modality in an approximate cross-modal number matching task. We presented different arrays of dowels (haptic stimuli) to 50 undergraduates and asked them to compare haptically perceived numerosity to two visually presented dot arrays. Participants chose which visually presented array matched the numerosity of the haptic stimulus. The distractor varied in number and displayed a random pattern, whereas the matching (target) dot array was either spatially identical or spatially randomized (to the haptic stimulus). We hypothesized that if a “numerosity” percept is based solely on number, neither spatially identical nor spatial congruence between the haptic and the visual target arrays would affect the accuracy in the task. However, results show significant processing advantages for targets with spatially identical patterns and, furthermore, that spatial congruency between haptic source and visual target facilitates performance. Our results show that spatial information was extracted from the haptic stimuli and influenced participants’ responses, which challenges the assumption that numerosity is represented in a truly abstract manner by filtering out any other stimulus features.

Generalized, cross-modal, and incrementing non-matching-to-sample in rats.

Same/different concept learning has been demonstrated in previous research in rats using matching- and non-matching-to-sample procedures with olfactory stimuli. In Experiment 1, rats were trained on the non-matching-to-sample procedure with either three-dimensional (3D plastic objects; n = 3) or olfactory (household spices, n = 5) stimuli, then tested for transfer to novel stimuli of the same, and then the alternate, modality. While all three rats trained with olfactory stimuli showed generalized non-matching to novel odors, only one rat learned the 3D relation and showed generalized transfer to novel objects. Importantly, in this rat the 3D non-matching relation then immediately transferred to odors. In contrast, rats trained with scents did not show transfer to novel3D stimuli until aftertraining with one or two 3D stimulus sets. In Experiment 2, four rats were trained on an incrementing non-matching-to-sample task featuring 3D plastic objects as stimuli (3D Span Task). Responses to session-novel stimuli resulted in reinforcement. Only two rats learned the 3D Span Task; one rat performed with high accuracy even with up to 17 session-novel objects in a session. While these findings emphasize the exceptional olfactory discrimination of rats relative to that with 3D/tactile/visual cues, they also show that relational learning can be demonstrated in another modality in this species. Further, the present study provides some evidence of cross-modal transfer of relational responding in rats.

Object Classification in SAR Imagery With Deep Cross-Modal Transfer Network

Object Classification in SAR Imagery With Deep Cross-Modal Transfer Network

Cross-Modal Transfer Learning Methods for Alzheimer's Disease Diagnosis.

In this paper we propose cross-modal transfer learning for Alzheimer's disease detection. We use positron emission tomography (PET) and magnetic resonance imaging (MRI) brain scans from ADNI to train convolutional neural networks (CNNs) on one modality and fine-tune it on the other modality. We start by showing that cross-modal transfer learning approaches outperform CNNs trained from scratch on a single modality. We then show that cross-modal transfer-learning also outperforms multimodal approaches using the same data.

An incremental cross-modal transfer learning method for gesture interaction

Gesture can be used as an important way for human–robot interaction, since it is able to give accurate and intuitive instructions to the robots. Various sensors can be used to capture gestures. We apply three different sensors that can provide different modalities in recognizing human gestures. Such data also owns its own statistical properties for the purpose of transfer learning: they own the same labeled data, but both the source and the validation data-sets have their own statistical distributions. To tackle the transfer learning problem across different sensors with such kind of data-sets, we propose a weighting method to adjust the probability distributions of the data, which results in a more faster convergence result. We further apply this method in a broad learning system, which has proven to be efficient to learn with the incremental learning capability. The results show that although these three sensors measure different parts of the body using different technologies, transfer learning is able to find out the weighting correlation among the data-sets. It also suggests that using the proposed transfer learning is able to adjust the data which has different distributions which may be similar to the physical correlation between different parts of the body in the context of giving gestures.

A supramodal and conceptual representation of subsecond time revealed with perceptual learning of temporal interval discrimination

Subsecond time perception has been frequently attributed to modality-specific timing mechanisms that would predict no cross-modal transfer of temporal perceptual learning. In fact, perceptual learning of temporal interval discrimination (TID) reportedly shows either no cross-modal transfer, or asymmetric transfer from audition to vision, but not vice versa. However, here we demonstrate complete cross-modal transfer of auditory and visual TID learning using a double training paradigm. Specifically, visual TID learning transfers to and optimizes auditory TID when the participants also receive exposure to the auditory temporal interval by practicing a functionally orthogonal near-threshold tone frequency discrimination task at the same trained interval. Auditory TID learning also transfers to and optimizes visual TID with additional practice of an orthogonal near-threshold visual contrast discrimination task at the same trained interval. Practicing these functionally orthogonal tasks per se has no impact on TID thresholds. We interpret the transfer results as indications of a supramodal representation of subsecond time. Moreover, because TID learning shows complete transfer between modalities with vastly different temporal precisions, the sub-second time presentation must be conceptual. Double training may refine this supramodal and conceptual subsecond time representation and connect it to a new sense to improve time perception.

Cross-modal Transfer of Valence or Arousal from Music to Word Targets in Affective Priming?

ABSTRACT This registered report considers how emotion induced in an auditory modality (music) can influence affective evaluations of visual stimuli (words). Specifically, it seeks to determine which emotional dimension is transferred across modalities – valence or arousal – or whether the transferred dimension depends on the focus of attention (feature-specific attention allocation). Two experiments were carried out. The first was an affective priming paradigm that will allow for the orthogonal manipulation of valence and arousal in both the words and music, alongside a manipulation to direct participants’ attention to either the valence or the arousal dimension. Secondly, a lexical decision task allowed cross-modal transfer of valence and arousal to be probed without the focus of participants’ attention being manipulated. Congruence effects were present in the affective priming task – valence was transferred in both the valence and arousal tasks, whereas arousal was transferred in the arousal task only. Contrary to predictions, the lexical decision task did not exhibit any congruence effects.

The cross-modal transfer of emotional information from voices to faces in 5-, 8- and 10-year-old children and adults: An eye-tracking study.

The present study examined the evolution of emotional cross-modal transfer throughout childhood compared to adulthood, using an experimental design first used with infants. We studied whether verbal children spontaneously look at emotional faces differently depending on the emotional voices previously heard, demonstrating a real intrinsic understanding of the emotion. Thus, sequences of emotional (happy vs. angry) cross-modal transfer were individually presented to 5-, 8- and 10-year-old children and adults. Spontaneous ocular behaviors toward the visual stimuli were recorded by eye-tracking. Results of the emotional cross-modal transfer suggested that participants looked spontaneously longer at the congruent face. However, this result was significantly revealed only as of age 8 with the happy voice and as of age 10 with the angry voice. Thus, the modulation of behavior indicators related to the control of the ability to extract amodal emotional information and spontaneously match the congruent information seems to increase with age and depends on the specific emotion presented. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Cross-Modal Transfer Learning From EEG to Functional Near-Infrared Spectroscopy for Classification Task in Brain-Computer Interface System.

The brain-computer interface (BCI) based on functional near-infrared spectroscopy (fNIRS) has received more and more attention due to its vast application potential in emotion recognition. However, the relatively insufficient investigation of the feature extraction algorithms limits its use in practice. In this article, to improve the performance of fNIRS-based BCI, we proposed a method named R-CSP-E, which introduces EEG signals when computing fNIRS signals’ features based on transfer learning and ensemble learning theory. In detail, we used the Independent Component Analysis (ICA) algorithm for the correspondence between the sources of the two signals. We then introduced the EEG signals when computing the spatial filter based on a modified Common Spatial Pattern (CSP) algorithm. Experimental results on public datasets show that the proposed method in this paper outperforms traditional methods without transfer. In general, the mean classification accuracy can be increased by up to 5%. To our knowledge, it is an innovation that we tried to apply transfer learning between EEG and fNIRS. Our study’s findings not only prove the potential of the transfer learning algorithm in cross-model brain-computer interface, but also offer a new and innovative perspective to research the hybrid brain-computer interface.

CT2US: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data

Accurate segmentation of kidney in ultrasound images is a vital procedure in clinical diagnosis and interventional operation. In recent years, deep learning technology has demonstrated promising prospects in medical image analysis. However, due to the inherent problems of ultrasound images, data with annotations are scarce and arduous to acquire, hampering the application of data-hungry deep learning methods. In this paper, we propose cross-modal transfer learning from computerized tomography (CT) to ultrasound (US) by leveraging annotated data in the CT modality. In particular, we adopt cycle generative adversarial network (CycleGAN) to synthesize US images from CT data and construct a transition dataset to mitigate the immense domain discrepancy between US and CT. Mainstream convolutional neural networks such as U-Net, U-Res, PSPNet, and DeepLab v3+ are pretrained on the transition dataset and then transferred to real US images. We first trained CNN models on a data set composed of 50 ultrasound images and validated them on a validation set composed of 30 ultrasound images. In addition, we selected 82 ultrasound images from another hospital to construct a cross-site data set to verify the generalization performance of the models. The experimental results show that with our proposed transfer learning strategy, the segmentation accuracy in dice similarity coefficient (DSC) reaches 0.853 for U-Net, 0.850 for U-Res, 0.826 for PSPNet and 0.827 for DeepLab v3+ on the cross-site test set. Compared with training from scratch, the accuracy improvement was 0.127, 0.097, 0.105 and 0.036 respectively. Our transfer learning strategy effectively improves the accuracy and generalization ability of ultrasound image segmentation model with limited training data.

Elder emotion classification through multimodal fusion of intermediate layers and cross-modal transfer learning.

The objective of the work is to develop an automated emotion recognition system specifically targeted to elderly people. A multi-modal system is developed which has integrated information from audio and video modalities. The database selected for experiments is ElderReact, which contains 1323 video clips of 3 to 8 s duration of people above the age of 50. Here, all the six available emotions Disgust, Anger, Fear, Happiness, Sadness and Surprise are considered. In order to develop an automated emotion recognition system for aged adults, we attempted different modeling techniques. Features are extracted, and neural network models with backpropagation are attempted for developing the models. Further, for the raw video model, transfer learning from pretrained networks is attempted. Convolutional neural network and long short-time memory-based models were taken by maintaining the continuity in time between the frames while capturing the emotions. For the audio model, cross-model transfer learning is applied. Both the models are combined by fusion of intermediate layers. The layers are selected through a grid-based search algorithm. The accuracy and F1-score show that the proposed approach is outperforming the state-of-the-art results. Classification of all the images shows a minimum relative improvement of 6.5% for happiness to a maximum of 46% increase for sadness over the baseline results.

Two-Stage Cross-Modality Transfer Learning Method for Military-Civilian SAR Ship Recognition

Military-civilian attribute recognition of ships in synthetic aperture radar (SAR) imagery plays an important role in marine surveillance. However, high-quality labeled data are hard to obtain for SAR ships, which hinder the development of deep learning models. Considering that models directly transferred from labeled optical images cannot achieve satisfactory performance for SAR applications due to the great discrepancy of different modalities, we propose a two-stage transfer learning method by combining the data-level and feature-level knowledge transfer. First, CycleGAN is adopted in the first stage to transfer the labeled optical image domain to the intermediate SAR-like image domain with the attribute labels. Then, a novel network called Domain Transfer using Adversarial learning and Metric learning (DTAM) is proposed to realize the task of military-civilian ship recognition by the domain adaption of the intermediate domain and the target SAR domain with joint adversarial learning and metric learning. To validate the proposed method, we establish a high-resolution SAR ship recognition dataset (HRSSRD), containing SAR and optical images of military and civilian ships. The experimental results show that the proposed two-stage architecture exhibits promising performance on the problem of SAR military-civilian ship recognition.

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

Impact of artificial intelligence on clinical radiography practice: Futuristic prospects in a low resource setting

ObjectivesCurrent trends in clinical radiography practice include the integration of artificial intelligence (AI) and related applications to improve patient care and enhance research. However, in low resource countries there are unique barriers to the process of AI integration. Using Ghana as a case study, this paper seeks to discuss the potential impact of AI on future radiographic practice in low-resource settings. The opportunities, challenges and the way forward to optimise the potential benefits of AI in future practice within these settings have been explored. Key findingsSome of the barriers to AI integration into radiographic practice relate to lack of regulatory and legal policy frameworks and limited resource availability including unreliable internet connectivity and low expert skillset. ConclusionThese barriers notwithstanding, AI presents a great potential to the growth of medical imaging and subsequently improving quality of healthcare delivery in the near future. For example, AI-enabled radiographer reporting has a potential to improving quality of healthcare, especially in low-resource settings like Ghana with an acute shortage of radiologists. In addition, futuristic AI-enabled advancements such as synthetic cross-modality transfer where images from one modality are used as a baseline to generate a corresponding image of another modality without the need for additional scanning will be of particular benefit in low-resource settings. Implications for practiceThe urgent need for inclusion of AI modules for the training of the radiographer of the future has been suggested. Recommendations for development of AI strategies by national societies and regulatory bodies will harmonise the implementation efforts. Finally, there is need for collaboration between clinical practitioners and academia to ensure that the future radiography workforce is well prepared for the AI-enabled clinical environment.

Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition

Speech emotion recognition is an important task with a wide range of applications. However, the progress of speech emotion recognition is limited by the lack of large, high-quality labeled speech datasets, due to the high annotation cost and the inherent ambiguity in emotion labels. The recent emergence of large-scale video data makes it possible to obtain massive, though unlabeled speech data. To exploit this unlabeled data, previous works have explored semi-supervised learning methods on various tasks. However, noisy pseudo-labels remain a challenge for these methods. In this work, to alleviate the above issue, we propose a new architecture that combines cross-modal knowledge transfer from visual to audio modality into our semi-supervised learning method with consistency regularization. We posit that introducing visual emotional knowledge by the cross-modal transfer method can increase the diversity and accuracy of pseudo-labels and improve the robustness of the model. To combine knowledge from cross-modal transfer and semi-supervised learning, we design two fusion algorithms, i.e. weighted fusion and consistent & random. Our experiments on CH-SIMS and IEMOCAP datasets show that our method can effectively use additional unlabeled audio-visual data to outperform state-of-the-art results.

Enhanced Audio Tagging via Multi- to Single-Modal Teacher-Student Mutual Learning

Recognizing ongoing events based on acoustic clues has been a critical yet challenging problem that has attracted significant research attention in recent years. Joint audio-visual analysis can improve the event detection accuracy but may not always be feasible as under many circumstances only audio recordings are available in real-world scenarios. To solve the challenges, we present a novel visual-assisted teacher-student mutual learning framework for robust sound event detection from audio recordings. Our model adopts a multi-modal teacher network based on both acoustic and visual clues, and a single-modal student network based on acoustic clues only. Conventional teacher-student learning performs unsatisfactorily for knowledge transfer from a multi-modality network to a single-modality network. We thus present a mutual learning framework by introducing a single-modal transfer loss and a cross-modal transfer loss to collaboratively learn the audio-visual correlations between the two networks. Our proposed solution takes the advantages of joint audio-visual analysis in training while maximizing the feasibility of the model in use cases. Our extensive experiments on the DCASE17 and the DCASE18 sound event detection datasets show that our proposed method outperforms the state-of-the-art audio tagging approaches.

Cross-Modal Transfer Following Auditory Task-Switching Training in Old Adults.

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory task-switching paradigm over four sessions (2880 total trials), to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks (active control) or not trained (passive control) allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level.

Cross-modal motion aftereffects transfer between vision and touch in early deaf adults

Previous research on early deafness has primarily focused on the behavioral and neural changes in the intact visual and tactile modalities. However, how early deafness changes the interplay of these two modalities is not well understood. In the current study, we investigated the effect of auditory deprivation on visuo-tactile interaction by measuring the cross-modal motion aftereffect. Consistent with previous findings, motion aftereffect transferred between vision and touch in a bidirectional manner in hearing participants. However, for deaf participants, the cross-modal transfer occurred only in the tactile-to-visual direction but not in the visual-to-tactile direction. This unidirectional cross-modal motion aftereffect found in the deaf participants could not be explained by unisensory motion aftereffect or discrimination threshold. The results suggest a reduced visual influence on tactile motion perception in early deaf individuals.