Modality Pairs Research Articles

Chromatic fusion: Generative multimodal neuroimaging data fusion provides multi-informed insights into schizophrenia.

This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. We apply our proposed framework, which disentangles multimodal data into private and shared sets of features from pairs of structural (sMRI), functional (sFNC and ICA), and diffusion MRI data (FA maps). With our approach, we find that heterogeneity in schizophrenia is potentially a function of modality pairs. Results show (1) schizophrenia is highly multimodal and includes changes in specific networks, (2) non-linear relationships with schizophrenia are observed when interpolating among shared latent dimensions, and (3) we observe a decrease in the modularity of functional connectivity and decreased visual-sensorimotor connectivity for schizophrenia patients for the FA-sFNC and sMRI-sFNC modality pairs, respectively. Additionally, our results generally indicate decreased fractional corpus callosum anisotropy, and decreased spatial ICA map and voxel-based morphometry strength in the superior frontal lobe as found in the FA-sFNC, sMRI-FA, and sMRI-ICA modality pair clusters. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data which we hope challenges the reader to think differently about how modalities interact.

Low-parameter supervised learning models can discriminate pseudoprogression and true progression in non-perfusion-based MRI.

Discrimination of pseudoprogression and true progression is one challenge to the treatment of malignant gliomas. Although some techniques such as circulating tumor DNA (ctDNA) and perfusion-weighted imaging (PWI) demonstrate promise in distinguishing PsP from TP, we investigate robust and replicable alternatives to distinguish the two entities based on more widely-available media. In this study, we use low-parametric supervised learning techniques based on geographically-weighted regression (GWR) to investigate the utility of both conventional MRI sequences as well as a diffusion-weighted sequence (apparent diffusion coefficient or ADC) in the discrimination of PsP v TP. GWR applied to MRI modality pairs is a unique approach for small sample sizes and is a novel approach in this arena. From our analysis, all modality pairs involving ADC maps, and those involving post-contrast T1 regressed onto T2 showed potential promise. This work on ADC data adds to a growing body of research suggesting the predictive benefits of ADC, and suggests further research on the relationships between post-contrast T1 and T2.Clinical relevance- Few studies have investigated predictive potential of conventional MRI and ADC to detect PsP. Our study adds to the growing research on the topic and presents a new perspective to research by exploiting the utility of ADC in PsP v TP distinction. In addition, our GWR methodology for low-parametric supervised computer vision models demonstrates a unique approach for image processing of small sample sizes.

TeFNA: Text-centered fusion network with crossmodal attention for multimodal sentiment analysis

Multimodal sentiment analysis (MSA), which goes beyond the analysis of texts to include other modalities such as audio and visual data, has attracted a significant amount of attention. An effective fusion of sentiment information in multiple modalities is key to improving the performance of MSA. However, aligning multiple modalities during the process of fusion faces challenges such as maintaining modal-specific information. This paper proposes a Text-centered Fusion Network with crossmodal Attention (TeFNA), a multimodal fusion network that uses crossmodal attention to model unaligned multimodal timing information. In particular, TeFNA employs a Text-Centered Aligned fusion method (TCA) that takes text modality as the primary modality to improve the representation of fusion features. In addition, TeFNA maximizes the mutual information between modality pairs to maintain task-related emotional information, thereby ensuring that the key information of modalities from input to fusion is preserved. The results of our comprehensive experiments on the multimodal datasets of CMU-MOSI and CMU-MOSEI show that our proposed model outperforms methods in terms of most metrics used.

ArithFusion: An Arithmetic Deep Model for Temporal Remote Sensing Image Fusion

Different satellite images may consist of variable numbers of channels which have different resolutions, and each satellite has a unique revisit period. For example, the Landsat-8 satellite images have 30 m resolution in their multispectral channels, the Sentinel-2 satellite images have 10 m resolution in the pan-sharp channel, and the National Agriculture Imagery Program (NAIP) aerial images have 1 m resolution. In this study, we propose a simple yet effective arithmetic deep model for multimodal temporal remote sensing image fusion. The proposed model takes both low- and high-resolution remote sensing images at t1 together with low-resolution images at a future time t2 from the same location as inputs and fuses them to generate high-resolution images for the same location at t2. We propose an arithmetic operation applied to the low-resolution images at the two time points in feature space to take care of temporal changes. We evaluated the proposed model on three modality pairs for multimodal temporal image fusion, including downsampled WorldView-2/original WorldView-2, Landsat-8/Sentinel-2, and Sentinel-2/NAIP. Experimental results show that our model outperforms traditional algorithms and recent deep learning-based models by large margins in most scenarios, achieving sharp fused images while appropriately addressing temporal changes.

Exploiting an Intermediate Latent Space between Photo and Sketch for Face Photo-Sketch Recognition.

The photo-sketch matching problem is challenging because the modality gap between a photo and a sketch is very large. This work features a novel approach to the use of an intermediate latent space between the two modalities that circumvents the problem of modality gap for face photo-sketch recognition. To set up a stable homogenous latent space between a photo and a sketch that is effective for matching, we utilize a bidirectional (photo → sketch and sketch → photo) collaborative synthesis network and equip the latent space with rich representation power. To provide rich representation power, we employ StyleGAN architectures, such as StyleGAN and StyleGAN2. The proposed latent space equipped with rich representation power enables us to conduct accurate matching because we can effectively align the distributions of the two modalities in the latent space. In addition, to resolve the problem of insufficient paired photo/sketch samples for training, we introduce a three-step training scheme. Extensive evaluation on a public composite face sketch database confirms superior performance of the proposed approach compared to existing state-of-the-art methods. The proposed methodology can be employed in matching other modality pairs.

Comparing ATN-T designation by tau PET visual reads, tau PET quantification, and CSF PTau181 across three cohorts.

To compare rates of tau biomarker positivity (T-status) per the 2018 Alzheimer's Disease (AD) Research Framework derived from [18F]flortaucipir (FTP) PET visual assessment, FTP quantification, and cerebrospinal fluid (CSF) phosphorylated Tau-181 (PTau181). We included 351 subjects with varying clinical diagnoses from three cohorts with available FTP PET and CSF PTau181 within 18months. T-status was derived from (1) FTP visual assessment by two blinded raters; (2) FTP standardized uptake value ratio (SUVR) quantification from a temporal meta-ROI (threshold: SUVR ≥1.27); and (3) Elecsys® Phospho-Tau (181P) CSF (Roche Diagnostics) concentrations (threshold: PTau181 ≥ 24.5pg/mL). FTP visual reads yielded the highest rates of T+, while T+ by SUVR increased progressively from cognitively normal (CN) through mild cognitive impairment (MCI) and AD dementia. T+ designation by CSF PTau181 was intermediate between FTP visual reads and SUVR values in CN, similar to SUVR in MCI, and lower in AD dementia. Concordance in T-status between modality pairs ranged from 68 to 76% and varied by clinical diagnosis, being highest in patients with AD dementia. In discriminating Aβ + MCI and AD subjects from healthy controls and non-AD participants, FTP visual assessment was most sensitive (0.96) but least specific (0.60). Specificity was highest with FTP SUVR (0.91) with sensitivity of 0.89. Sensitivity (0.73) and specificity (0.72) were balanced for PTau181. The choice of tau biomarker may differ by disease stage and research goals that seek to maximize sensitivity or specificity. Visual interpretations of tau PET enhance sensitivity compared to quantification alone, particularly in early disease stages.

Identifying a supramodal language network in human brain with individual fingerprint

Where is human language processed in the brain independent of its form? We addressed this issue by analyzing the cortical responses to spoken, written and signed sentences at the level of individual subjects. By applying a novel fingerprinting method based on the distributed pattern of brain activity, we identified a left-lateralized network composed by the superior temporal gyrus/sulcus (STG/STS), inferior frontal gyrus (IFG), precentral gyrus/sulcus (PCG/PCS), and supplementary motor area (SMA). In these regions, the local distributed activity pattern induced by any of the three language modalities can predict the activity pattern induced by the other two modalities, and such cross-modal prediction is individual-specific. The prediction is successful for speech-sign bilinguals across all possible modality pairs, but fails for monolinguals across sign-involved pairs. In comparison, conventional group-mean focused analysis detects shared cortical activations across modalities only in the STG, PCG/PCS and SMA, and the shared activations were found in both groups. This study reveals the core language system in the brain that is shared by spoken, written and signed language, and demonstrates that it is possible and desirable to utilize the information of individual differences for functional brain mapping.

Exercise-Induced Changes of Multimodal Interactions Within the Autonomic Nervous Network.

Physical exercise has been shown to modulate activity within the autonomic nervous system (ANS). Considering physical exercise as a holistic stimulus on the nervous system and specifically the ANS, uni- and multimodal analysis tools were applied to characterize centrally driven interactions and control of ANS functions. Nineteen young and physically active participants performed treadmill tests at individually determined moderate and high intensities. Continuous electrodermal activity (EDA), heart rate (HR), and skin temperature at wrist (Temp) were recorded by wireless multisensor devices (Empatica® E4, Milan, Italy) before and 30 min after exercise. Artifact-free continuous 3 min intervals were analyzed. For unimodal analysis, mean values were calculated, for bimodal and multimodal analysis canonical correlation analysis (CCA) was performed. Unimodal results indicate that physical exercise affects ANS activity. More specifically, Temp increased due to physical activity (moderate intensity: from 34.15°C to 35.34°C and high intensity: from 34.11°C to 35.09°C). HR increased more for the high (from 60.76 bpm to 79.89 bpm) than for the moderate (from 64.81 bpm to 70.83 bpm) intensity. EDA was higher for the high (pre: 8.06 μS and post: 9.37 μS) than for the moderate (pre: 4.31 μS and post: 3.91 μS) intensity. Bimodal analyses revealed high variations in correlations before exercise. The overall correlation coefficient showed varying correlations in pretest measures for all modality pairs (EDA-HR, HR-Temp, Temp-EDA at moderate: 0.831, 0.998, 0.921 and high: 0.706, 0, 0.578). After exercising at moderate intensity coefficients changed little (0.828, 0.744, 0.994), but increased substantially for all modality pairs after exercising at high intensity (0.976, 0.898, 0.926). Multimodal analysis confirmed bimodal results. Exercise-induced changes in ANS activity can be found in multiple ANS modalities as well as in their interactions. Those changes are intensity-specific: with higher intensity the interactions increase. Canonical correlations between different ANS modalities may therefore offer a feasible approach to determine exercise induced modulations of ANS activity.

The Effects of Masking on the Detection of Alarms in Close Temporal Proximity

The Effects of Masking on the Detection of Alarms in Close Temporal Proximity

Guiding multimodal registration with learned optimization updates.

In this paper, we address the multimodal registration problem from a novel perspective, aiming to predict the transformation aligning images directly from their visual appearance. We formulate the prediction as a supervised regression task, with joint image descriptors as input and the output are the parameters of the transformation that guide the moving image towards alignment. We model the joint local appearance with context aware descriptors that capture both local and global cues simultaneously in the two modalities, while the regression function is based on the gradient boosted trees method capable of handling the very large contextual feature space. The good properties of our predictions allow us to couple them with a simple gradient-based optimization for the final registration. Our approach can be applied to any transformation parametrization as well as a broad range of modality pairs. Our method learns the relationship between the intensity distributions of a pair of modalities by using prior knowledge in the form of a small training set of aligned image pairs (in the order of 1-5 in our experiments). We demonstrate the flexibility and generality of our method by evaluating its performance on a variety of multimodal imaging pairs obtained from two publicly available datasets, RIRE (brain MR, CT and PET) and IXI (brain MR). We also show results for the very challenging deformable registration of Intravascular Ultrasound and Histology images. In these experiments, our approach has a larger capture range when compared to other state-of-the-art methods, while improving registration accuracy in complex cases.

EP 27. Modulatory effect of motor cortex stimulation on semantic priming

Background Processing hand and foot verbs has been shown to activate the respective motor areas and to elicit beta desynchronization similar to movement execution ( van Elk et al., 2010 , Niccolai et al., 2014 ). Above that, oscillatory brain responses have shown that motor execution interferes with simultaneous verb processing ( Klepp et al., 2015 ). Objectives By exploiting the up- and down-regulation that anodal and cathodal transcranial direct current stimulation (tDCS) respectively exert on motor cortical excitability ( Nitsche and Paulus, 2000 ), we aimed at modulating semantic priming of action verbs. We expected that online anodal stimulation of the left hand motor cortex would slow down the processing of hand verbs by increasing cortical excitability and inducing the previously observed interference effect. The opposite outcome was expected for cathodal stimulation. Sham stimulation was used as a control condition in a double-blind within-subjects design. Methods The stimuli set consisted of disyllabic hand, foot, and non-body (abstract) verbs followed by shapes with rounded or pointed corners. Twenty healthy participants (10 females) were required to read the verbs and to perform a go/no-go task: (a) to respond only in trials with concrete verbs, (b) by pressing a manual button if the shape prompt had rounded/pointed corners and a foot pedal if the shape had pointed/rounded corners. Results Results showed a semantic priming effect consisting in significantly shorter reaction times to congruent verb type – response modality pairs. Cathodal stimulation resulted in less wrong answers for incongruent pairs compared to the sham condition. Responses to shapes following hand verbs were significantly faster in the cathodal compared to the anodal condition when responding with the hand. Conversely, button responses to shapes following hand and foot verbs in the anodal condition were slower compared to the sham condition. Conclusions Cathodal stimulation of the hand motor area appears to selectively improve processing of hand verbs. Anodal stimulation showed an interference effect unrelated to verb type, probably depending on modulation of both hand verbs processing and hand responses. As button responses were overall faster than pedal responses, the lack of modulation of the latter suggests either that the effect of tDCS on word processing is short-timed or that tDCS influenced both verb processing and hand responses.

Statistically Optimal Multisensory Cue Integration: A Practical Tutorial.

Humans combine redundant multisensory estimates into a coherent multimodal percept. Experiments in cue integration have shown for many modality pairs and perceptual tasks that multisensory information is fused in a statistically optimal manner: observers take the unimodal sensory reliability into consideration when performing perceptual judgments. They combine the senses according to the rules of Maximum Likelihood Estimation to maximize overall perceptual precision. This tutorial explains in an accessible manner how to design optimal cue integration experiments and how to analyse the results from these experiments to test whether humans follow the predictions of the optimal cue integration model. The tutorial is meant for novices in multisensory integration and requires very little training in formal models and psychophysical methods. For each step in the experimental design and analysis, rules of thumb and practical examples are provided. We also publish Matlab code for an example experiment on cue integration and a Matlab toolbox for data analysis that accompanies the tutorial online. This way, readers can learn about the techniques by trying them out themselves. We hope to provide readers with the tools necessary to design their own experiments on optimal cue integration and enable them to take part in explaining when, why and how humans combine multisensory information optimally.

Training Modalities and Self-Confidence Building in Performance of Life-Saving Procedures

Physicians and paramedics in the Israel Defense Forces are trained to perform advanced medical procedures using standardized training modalities, such as manikins. We studied the association of experience using these training modalities with self-reported confidence in procedure performance. Providers were sent a questionnaire regarding their experience with and self-confidence levels for performing endotracheal intubation, cricothyroidotomy, needle chest decompression, tube thoracostomy, and intraosseous infusion. Provider level (physician or paramedic) and gender were associated with reported self-confidence levels. Manikin and supervised and unsupervised patient experience exhibited positive associations with self-confidence, but (animal) model experience did not. For many procedure-training modality pairs, we identified a plateau level above which additional experience was minimally associated with an increase in self-confidence. Among military advanced life support providers, self-confidence levels in procedure performance are positively associated with experience gained from manikins and supervised and unsupervised patient application. We were not able to demonstrate a clear benefit of an animal model in increasing self-confidence. A plateau was generally identified, indicating decreased benefit from the use of a particular training modality for a particular procedure. Modifying training regimens in light of these findings may help maximize the self-confidence of advanced life support providers more efficiently.

Biomedical Imaging Modality Classification Using Combined Visual Features and Textual Terms

We describe an approach for the automatic modality classification in medical image retrieval task of the 2010 CLEF cross-language image retrieval campaign (ImageCLEF). This paper is focused on the process of feature extraction from medical images and fuses the different extracted visual features and textual feature for modality classification. To extract visual features from the images, we used histogram descriptor of edge, gray, or color intensity and block-based variation as global features and SIFT histogram as local feature. For textual feature of image representation, the binary histogram of some predefined vocabulary words from image captions is used. Then, we combine the different features using normalized kernel functions for SVM classification. Furthermore, for some easy misclassified modality pairs such as CT and MR or PET and NM modalities, a local classifier is used for distinguishing samples in the pair modality to improve performance. The proposed strategy is evaluated with the provided modality dataset by ImageCLEF 2010.

Using correspondence analysis with a large set of transition matrices. Example with eye movement data and fuzzy space windowing

This paper considers the data analysis process as a chronology of action triplets in which both the first and the third actions of each single triplet require a human intervention and the in-between action requires a computational approach. This process is used for the investigation of time units where each unit can be described using a qualitative variable with either crisp or fuzzy features. Then, each time, a unit set is characterized via a transitional matrix (TM). If in most cases only one TM is plotted with the primary objective of generating a Markov process model featuring S behavior states, this paper therefore considers the case where several TMs are used, e.g. several individuals yield several matrices in an experimental design. To investigate the transition data sets, we suggest using Correspondence Analysis (CA) via an input table containing the values from these TMs, each row including the S*S values of each TM. In addition, several new TMs can be computed from the initial TM, for instance one overall table for each individual or for each experimental condition. Within CA, these additional TMs can be considered supplementary row-points to demonstrate to what extent individual behaviors are similar. To illustrate this use of CA, we consider data from 1) simulated signals and 2) a driving vigilance study during which eye movements were recorded and characterized though eye fixation TM with S fuzzy areas of interest. The benefits and drawbacks of our approach are discussed and a comparison is performed with two hierarchical clustering processes used respectively with the transition matrices and modality pairs.

Crossmodal Direct Priming between Vision and Touch

This study examined whether a letter prime presented in one modality facilitate a subsequent response to the same letter target presented in the other modality. A prime and a target were presented either in vision or touch. The task was a go/no-go task, in which subjects were required to press a key as quickly as possible if the target was either the letter L or T (go trials) and refrain from responding if the target was a 5-dot pattern (no-go trials). Results from two experiments showed facilitatory priming effects in both vision-touch (VT) and touch-vision (TV) modality pairs at both short and long target onsets. This suggests that the visuotactile crossmodal priming effects endure for a certain period of time and that tactile stimuli take time to be processed.

Modality-repetition and attentional effects on reflex blinking in infants and adults

The ability of a brief prepulse to inhibit reflexively elicited startle is a robust phenomenon in adults of several species but is weak or absent in infants. The possibility that afferent processing of the prepulse is inadequate in infants was tested by assessing the integrity of two other types of reflex modulation that occur in modality-specific paths, that is, attenuation by repetition of same-modality stimuli and enhancement by modality-selective attention. The reflex, measured by electromyographic activity of the muscle controlling blink, was elicited by intense light flashes or noise bursts preceded by brief acoustic or visual prepulses and delivered during attention-directing acoustic or visual foregrounds that evoked cardiac deceleration. Modality-repetition effects were evidenced by smaller peak blink magnitude and longer latency of blinking to same than to different modality pairs and did not differ as a function of age (4-month-old infants or college students). Attention effects were also seen in both infants and adults and, in accord with previous findings, were evident in magnitude for infants and latency for adults. Thus, immaturity in paths mediating these effects could not explain delayed development of prepulse inhibition. Other possibilities include delayed maturation of extrinsic inhibition or of transient-processing systems.