Intermodality Correlation Research Articles

Graph-based Consistent Reconstruction and Alignment for imbalanced text-image person re-identification

Text-image person re-identification (TIReID) has emerged as a versatile approach for retrieving target pedestrians using textual descriptions. However, current TIReID research has been overly idealistic and has overlooked the issues of data incompleteness and modal imbalance in real-world application scenarios. Therefore, in this paper, we propose imbalanced text-image person re-identification (ITIReID) to address these problems. In comparison to TIReID, ITIReID contains a larger proportion of unimodal data, which leads to modal imbalance. The setting of ITIReID is more aligned with real-world scenarios, and studying ITIReID can expand the application scalability of TIReID. We propose a Graph-based Consistent Reconstruction and Alignment framework (GCRA), for ITIReID, which achieves modal balance by completing missing modality features for training implementation. By treating the accessible modality features as graph nodes, GCRA firstly builds an adjacency graph where a new semantic distance that establishes semantic relevance between nodes by comprehensively measuring both intra-modality and inter-modality correlation, serves as the measurement of graph’s edges. GCRA further reconstructs the missing nodes - thus re-establishing missing modality features - using existing nodes connected with high semantic relevance. To ensure the reliability and effectiveness of reconstructed features, we propose a proxy-based identity constraint and a reconstruction constraint. In addition, to enable effective semantic alignment using both the reconstructed features and original features, we introduce a cross-modal semantic constraint. Extensive experiments demonstrate that GCRA can effectively handle issues of data incompleteness and modal imbalance, exhibiting its effectiveness and superiority.

Diagnostic performance of prospective same-day 18F-FDG PET/MRI and 18F-FDG PET/CT in the staging and response assessment of lymphoma

BackgroundAccurate staging and response assessment are essential for prognosis and to guide treatment in patients with lymphoma. The aim of this study was to compare the diagnostic performance of FDG PET/MRI versus FDG PET/CT in adult patients with newly diagnosed Hodgkin and Non- Hodgkin lymphoma.MethodsIn this single centre study, 50 patients were prospectively recruited. FDG PET/MRI was performed after staging FDG PET/CT using a single injection of 18F-FDG. Patients were invited to complete same-day FDG PET/MRI with FDG PET/CT at interim and end of treatment response assessments. Performance was assessed using PET/CT as the reference standard for disease site identification, staging, response assessment with Deauville score and concordance in metabolic activity.ResultsStaging assessment showed perfect agreement (κ = 1.0, P = 0) between PET/MRI and PET/CT using Ann Arbor staging. There was excellent intermodality correlation with disease site identification at staging (κ = 0.976, P < 0.001) with FDG PET/MRI sensitivity of 96% (95% CI, 94–98%) and specificity of 100% (95% CI, 99–100%). There was good correlation of disease site identification at interim assessment (κ = 0.819, P < 0.001) and excellent correlation at end-of-treatment assessment (κ = 1.0, P < 0.001). Intermodality agreement for Deauville scores was good at interim assessment (κ = 0.808, P < 0.001) and excellent at end-of-treatment assessment (κ = 1.0, P = 0). There was good–excellent concordance in SUV max and mean between modalities across timepoints. Minimum calculated radiation patient effective dose saving was 54% between the two modalities per scan.ConclusionWith high concordance in disease site identification, staging and response assessment, PET/MR is a potentially viable alternative to PET/CT in lymphoma that minimises radiation exposure.

Visible and Infrared Object Tracking via Convolution-Transformer Network With Joint Multimodal Feature Learning

The existing Transformer-based RGBT tracker mainly focus on the enhancement of features extracted by Convolutional Neural Network (CNN). The potential of Transformer in representation learning remains under-explored. In this paper, we propose a Convolution-Transformer network with joint multimodal feature learning, in which both representation learning and feature fusion leverage Transformer. Specifically, we use the multi-branch Convolution-Transformer feature extraction network to process the extraction task of local modality-independent features and global modality-shared features respectively. Several simplified Transformer encoder layers form the Transformer backbone network, which is more suitable for the real-time object tracking. Besides, we found that inter-modality correlation is an important factor for modality interactions and mutual exploitation. Therefore, we propose a Joint Multimodal Feature Learning (JMFL) module, which uses cross-attention to capture the dependencies of cross-modal and enhance multimodal fusion by bidirectional guidance of multimodal information. The proposed method is fully experimented on two large benchmark datasets and compared with some current well-performing methods. The experimental results show that the proposed method performs well in terms of tracking accuracy and speed.

Effective Exploitation of Macroeconomic Indicators for Stock Direction Classification Using the Multimodal Fusion Transformer

An enormous ripple effect can occur in financial data mining if it accurately predicts stock prices. However, predicting stock prices using only stock price data is difficult because of the random nature of stock price data. This paper attempts to fuse data to solve the stock price prediction problem. The following data affecting the stock price are added to the proposed method as an additional modality: macroeconomic indicators and the months and day of the week. The multimodal early fusion method is used, which learns the intermodality correlation of features. The proposed model in this paper outperformed the comparison models and achieved statistically significant results. Specifically, 27 out of 50 stocks achieved higher classification accuracy than the comparative model. In addition, the in-depth analysis indicates that the early fusion strategy achieved better classification accuracy in 30 of 50 datasets than the late fusion strategy for stock price prediction.

The use of mixed reality technology for the objective assessment of clinical skills: a validation study

BackgroundMixed Reality technology may provide many advantages over traditional teaching methods. Despite its potential, the technology has yet to be used for the formal assessment of clinical competency. This study sought to collect validity evidence and assess the feasibility of using the HoloLens 2 mixed reality headset for the conduct and augmentation of Objective Structured Clinical Examinations (OSCEs).MethodsA prospective cohort study was conducted to compare the assessment of undergraduate medical students undertaking OSCEs via HoloLens 2 live (HLL) and recorded (HLR), and gold-standard in-person (IP) methods. An augmented mixed reality scenario was also assessed.ResultsThirteen undergraduate participants completed a total of 65 OSCE stations. Overall inter-modality correlation was 0.81 (p = 0.01), 0.98 (p = 0.01) and 0.82 (p = 0.01) for IP vs. HLL, HLL vs. HLR and IP vs. HLR respectively. Skill based correlations for IP vs. HLR were assessed for history taking (0.82, p = 0.01), clinical examination (0.81, p = 0.01), procedural (0.88, p = 0.01) and clinical skills (0.92, p = 0.01), and assessment of a virtual mixed reality patient (0.74, p = 0.01). The HoloLens device was deemed to be usable and practical (Standard Usability Scale (SUS) score = 51.5), and the technology was thought to deliver greater flexibility and convenience, and have the potential to expand and enhance assessment opportunities.ConclusionsHoloLens 2 is comparable to traditional in-person examination of undergraduate medical students for both live and recorded assessments, and therefore is a valid and robust method for objectively assessing performance. The technology is in its infancy, and users need to develop confidence in its usability and reliability as an assessment tool. However, the potential to integrate additional functionality including holographic content, automated tracking and data analysis, and to facilitate remote assessment may allow the technology to enhance, expand and standardise examinations across a range of educational contexts.

Object-Level Visual-Text Correlation Graph Hashing for Unsupervised Cross-Modal Retrieval.

The core of cross-modal hashing methods is to map high dimensional features into binary hash codes, which can then efficiently utilize the Hamming distance metric to enhance retrieval efficiency. Recent development emphasizes the advantages of the unsupervised cross-modal hashing technique, since it only relies on relevant information of the paired data, making it more applicable to real-world applications. However, two problems, that is intro-modality correlation and inter-modality correlation, still have not been fully considered. Intra-modality correlation describes the complex overall concept of a single modality and provides semantic relevance for retrieval tasks, while inter-modality correction refers to the relationship between different modalities. From our observation and hypothesis, the dependency relationship within the modality and between different modalities can be constructed at the object level, which can further improve cross-modal hashing retrieval accuracy. To this end, we propose a Visual-textful Correlation Graph Hashing (OVCGH) approach to mine the fine-grained object-level similarity in cross-modal data while suppressing noise interference. Specifically, a novel intra-modality correlation graph is designed to learn graph-level representations of different modalities, obtaining the dependency relationship of the image region to image region and the tag to tag in an unsupervised manner. Then, we design a visual-text dependency building module that can capture correlation semantic information between different modalities by modeling the dependency relationship between image object region and text tag. Extensive experiments on two widely used datasets verify the effectiveness of our proposed approach.

Reproducibility of 4D cardiac computed tomography feature tracking myocardial strain and comparison against speckle-tracking echocardiography in patients with severe aortic stenosis

BackgroundMyocardial strain is an established parameter for the assessment of cardiac function and routinely derived from speckle tracking echocardiography (STE). Novel post-processing tools allow deformation imaging also by 4D cardiac computed tomography angiography (CCT). This retrospective study aims to analyze the reproducibility of CCT strain and compare it to that of STE. MethodsLeft (LV) and right ventricular (RV), and left atrial (LA) ejection fraction (EF), dimensions, global longitudinal (GLS), circumferential (GCS) and radial strain (GRS) were determined by STE and CCT feature tracking in consecutive patients with severe aortic stenosis evaluated for transcatheter aortic valve implantation. Results106 patients (mean age 79.9 ​± ​7.8, 44.3% females) underwent CCT at a median of 3 days (IQR 0–28 days) after STE. In CCT, strain measures showed good to excellent reproducibility (intra- and inter-reader intraclass correlation coefficient ≥0.75) consistently in the LV, RV and LA. In STE, only LV GLS and LA GLS yielded good reproducibility, whereas LV GCS and LV GRS showed moderate, and RV GLS and free wall longitudinal strain (FWLS) poor reproducibility. Agreement between CCT and STE was strong for LV GLS only, while other strain features displayed moderate (LV GCS, LA GLS) or weak (LV GRS, RV GLS and FWLS) inter-modality correlation. ConclusionLV, RV and LA CCT strain assessments were highly reproducible. While a strong agreement to STE was found for LV GLS, inter-modality correlation was moderate or weak for LV GCS, LV GRS, and RV and LA longitudinal strain, possibly related to poor reproducibility of STE measurements.

Comparison of left atrial strain by feature-tracking cardiac magnetic resonance with speckle-tracking transthoracic echocardiography.

Left atrial (LA) strain is a novel non-invasive parameter for assessing LA hemodynamics and function. We sought to compare the intermodality differences between transthoracic echocardiography (TTE) and cardiac magnetic resonance (CMR) derived LA strain, as well as reproducibility of strain measurements. We evaluated 70 subjects (mean age 42.1 ± 17 years, 44% males) with no significant cardiovascular disease who underwent both CMR and TTE within 6 months of each other. LA strain measurements i.e. reservoir strain (ƐR), conduit strain (ƐCD), and contractile strain (ƐCT), were compared using speckle-tracking echocardiography (STE) and CMR feature tracking (CMR-FT). Correlation and systematic bias between modalities was evaluated using intraclass correlation coefficient (ICC) and proportional bias. TTE was performed before CMR with a median duration of 33 days (IQR 14-69 days). ICC for ƐR, ƐCT, ƐCD was 0.66 (95% CI 0.44-0.79), 0.63 (95% CI 0.4-0.77) and 0.56 (95% CI 0.3-0.73) respectively. There was evidence of systematic bias between modalities, on average LA volume was found to be 19% higher on CMR than TTE. Strain values were also higher by CMR-FT compared to STE with mean difference of 9.9 ± 12 (26.1%), 3.1 ± 5.5 (21.9%), 4.0 ± 9.9 (16.6%) for ƐR, ƐCT and ƐCD respectively. Regression showed proportional bias for both ƐR, and ƐCT (beta 0.76, 0.54 respectively; P < 0.0001). There were modest differences in intraobserver reproducibility between both modalities with better reproducibility for STE compared to CMR-FT. There was a modest intermodality correlation between STE and CMR-FT derived LA strain components. There were systematic differences and proportional bias in measurements between modalities. These differences should be considered when interpreting LA strain using either modality.

Utilizing a mobile platform to obtain quantitative measurements of exophthalmos

ABSTRACT Purpose This study introduces a novel exophthalmometry method utilizing a mobile platform to obtain calibrated quantitative measurements of globe position and compares its reliability to Hertel exophthalmometry. Methods A prospective, comparative study included 50 patients (99 eyes) for a total of 594 mobile measurements. Healthy individuals from the community in Augusta, Georgia, and individuals from a routine oculoplastics clinic at Mayo Clinic in Rochester, Minnesota, were included to represent variety within the population. The main outcome was the inter-modality correlation and agreement between Hertel and mobile exophthalmometry, and the intra-observer and inter-observer reliability with repeated mobile measurements. Results There was no significant difference in the measurements obtained by Hertel exophthalmometry and mobile exophthalmometry. There was a strong linear correlation between Hertel and mobile exophthalmometry with a Pearson Correlation Coefficient of 0.910 and 0.888 for the right and left eyes, respectively (p < .001, 2-tailed). Bland–Altman plot analysis showed excellent agreement between the two modalities. The mobile platform demonstrated high intra-observer reliability with a Cronbach’s alpha of 0.992 and 0.985 for the right and left eyes. An intraclass correlation coefficient of 0.992 (95% CI: 0.987–0.995) for the right eye and 0.986 (95% CI: 0.978–0.991) for the left eye demonstrated excellent reliability between observers. Conclusions Mobile exophthalmometry may be a promising tool for obtaining calibrated quantitative measurements of globe position for situations in which Hertel exophthalmometry is not available. The strong correlation and excellent agreement between Hertel and mobile measurement suggest that mobile exophthalmometry can yield reliable and accurate measurements.

Polygenic Hazard Score Associated Multimodal Brain Networks Along the Alzheimer's Disease Continuum.

BackgroundLate-onset Alzheimer’s disease (AD) is a polygenic neurodegenerative disease. Identifying the neuroimaging phenotypes behind the genetic predisposition of AD is critical to the understanding of AD pathogenesis. Two major questions which previous studies have led to are: (1) should the general “polygenic hazard score” (PHS) be a good choice to identify the individual genetic risk for AD; and (2) should researchers also include inter-modality relationships in the analyses considering these may provide complementary information about the AD etiology.MethodsWe collected 88 healthy controls, 77 patients with mild cognitive impairment (MCI), and 22 AD patients to simulate the AD continuum included from the ADNI database. PHS-guided multimodal fusion was used to investigate the impact of PHS on multimodal brain networks in AD-continuum by maximizing both inter-modality association and reference-modality correlation. Fractional amplitude of low frequency fluctuations, gray matter (GM) volume, and amyloid standard uptake value ratios were included as neuroimaging features. Eventually, the changes in neuroimaging features along AD continuum were investigated, and relationships between cognitive performance and identified PHS associated multimodal components were established.ResultsWe found that PHS was associated with multimodal brain networks, which showed different functional and structural impairments under increased amyloid deposits. Notably, along with AD progression, functional impairment occurred before GM atrophy, amyloid deposition started from the MCI stage and progressively increased throughout the disease continuum.ConclusionPHS is associated with multi-facets of brain impairments along the AD continuum, including cognitive dysfunction, pathological deposition, which might underpin the AD pathogenesis.

Retro mode illumination for detecting and quantifying the area of geographic atrophy in non-neovascular age-related macular degeneration.

To evaluate the ability of retro mode illumination imaging for quantifying atrophy compared to confocal color fundus photography (c-CFP), green light fundus autofluorescence (G-FAF), blue light fundus autofluorescence (B-FAF) using the scanning laser ophthalmoscope (SLO) Mirante device by Nidek (Nidek Co., Ltd, Gamogori, Japan). Eyes with clinical evidence of geographic atrophy (GA) associated with non-neovascular age-related macular degeneration, evaluated at the Doheny Eye Centers-UCLA and Hospital Sacco Milan, were included in this prospective, cross-sectional study. All eyes were imaged with multiple retinal imaging modalities using the SLO Nidek Mirante device: c-CFP, G-FAF, B-FAF, retro mode illumination deviated-right (RMDR), and deviated-left (RMDL). Masked graders measured the GA lesion on each modality and inter-modality and inter-grader repeatability were assessed. The mean (SD) area of GA measured 9.76 (3.82) mm2, 9.75 (3.91) mm2, 9.76 (3.92) mm2, 9.82 (3.87) mm2, and 9.81 (3.86) mm2 using c-CFP, G-FAF, B-FAF, RMDR, and RMDL, respectively (p = 0.2). Inter-modality correlation was high (Pearson's r > 0.9 and p < 0.0001). Agreement between graders was excellent for all modalities. Retro mode imaging demonstrated good agreement for measuring GA compared to other imaging modalities, with a high level of repeatability. Given that retro mode imaging uses infrared light and is comfortable, it may prove to be a useful tool for the assessment of GA in the clinic.

Non-invasive three-dimensional thickness analysis of oral epithelium based on optical coherence tomography—development and diagnostic performance

ObjectivesEvaluating structural changes in oral epithelium can assist with the diagnosis of cancerous lesions. Two-dimensional (2D) non-invasive optical coherence tomography (OCT) is an established technique for this purpose. The objective of this study was to develop and test the diagnostic accuracy of a three-dimensional (3D) evaluation method. MethodsThe oral lip mucosa of 10 healthy volunteers was scanned using an 870-nm spectral-domain OCT device (SD-OCT) with enhanced depth imaging (EDI). Four raters semi-automatically segmented the epithelial layer twice. Thus, eighty 3D datasets were created and analyzed for epithelial thickness. To provide a reference standard for comparison, the raters took cross-sectional 2D measurements at representative sites. The correlation between the 2D and 3D measurements, as well as intra- and inter-rater reliability, were analyzed using intraclass correlation coefficients (ICC). ResultsMean epithelial thickness was 280 ± 64μm (range 178–500 μm) and 268 ± 49μm (range 163–425 μm) for the 2D and 3D analysis, respectively. The inter-modality correlation of the thickness values was good (ICC: 0.76 [0.626–0.846]), indicating that 3D analysis of epithelial thickness provides valid results. Intra-rater and inter-rater reliability were good (3D analysis) and excellent (2D analysis), suggesting high reproducibility. ConclusionsDiagnostic accuracy was high for the developed 3D analysis of oral epithelia using non-invasive, radiation-free OCT imaging. Clinical significanceThis new 3D technique could potentially be used to improve time-efficiency and quality in the diagnosis of epithelial lesions compared with the 2D reference standard.

QUANTIFICATION OF PULMONARY PATHOLOGY IN CYSTIC FIBROSIS-COMPARISON BETWEEN DIGITAL CHEST TOMOSYNTHESIS AND COMPUTED TOMOGRAPHY.

Purpose: Digital tomosynthesis (DTS) is currently undergoing validation for potential clinical implications. The aim of this study was to investigate the potential for DTS as a low-dose alternative to computed tomography (CT) in imaging of pulmonary pathology in patients with cystic fibrosis (CF). Methods: DTS and CT were performed as part of the routine triannual follow-up in 31 CF patients. Extent of disease was quantified according to modality-specific scoring systems. Statistical analysis included Spearman’s rank correlation coefficient (r) and Krippendorff’s alpha (α). Major findings: The median effective dose was 0.14 for DTS and 2.68 for CT. Intermodality correlation was very strong for total score and the subscores regarding bronchiectasis and bronchial wall-thickening (r = 0.82–0.91, P < 0.01). Interobserver reliability was high for total score, bronchiectasis and mucus plugging (α = 0.83–0.93) in DTS. Conclusion: Chest tomosynthesis could be a low-dose alternative to CT in quantitative estimation of structural lung disease in CF.

Deep Learning-Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors.

BackgroundMultimodal wearable technologies have brought forward wide possibilities in human activity recognition, and more specifically personalized monitoring of eating habits. The emerging challenge now is the selection of most discriminative information from high-dimensional data collected from multiple sources. The available fusion algorithms with their complex structure are poorly adopted to the computationally constrained environment which requires integrating information directly at the source. As a result, more simple low-level fusion methods are needed.ObjectiveIn the absence of a data combining process, the cost of directly applying high-dimensional raw data to a deep classifier would be computationally expensive with regard to the response time, energy consumption, and memory requirement. Taking this into account, we aimed to develop a data fusion technique in a computationally efficient way to achieve a more comprehensive insight of human activity dynamics in a lower dimension. The major objective was considering statistical dependency of multisensory data and exploring intermodality correlation patterns for different activities.MethodsIn this technique, the information in time (regardless of the number of sources) is transformed into a 2D space that facilitates classification of eating episodes from others. This is based on a hypothesis that data captured by various sensors are statistically associated with each other and the covariance matrix of all these signals has a unique distribution correlated with each activity which can be encoded on a contour representation. These representations are then used as input of a deep model to learn specific patterns associated with specific activity.ResultsIn order to show the generalizability of the proposed fusion algorithm, 2 different scenarios were taken into account. These scenarios were different in terms of temporal segment size, type of activity, wearable device, subjects, and deep learning architecture. The first scenario used a data set in which a single participant performed a limited number of activities while wearing the Empatica E4 wristband. In the second scenario, a data set related to the activities of daily living was used where 10 different participants wore inertial measurement units while performing a more complex set of activities. The precision metric obtained from leave-one-subject-out cross-validation for the second scenario reached 0.803. The impact of missing data on performance degradation was also evaluated.ConclusionsTo conclude, the proposed fusion technique provides the possibility of embedding joint variability information over different modalities in just a single 2D representation which results in obtaining a more global view of different aspects of daily human activities at hand, and yet preserving the desired performance level in activity recognition.

Validation of standing cone beam computed tomography for diagnosing subchondral fetlock pathology in the Thoroughbred racehorse.

Subchondral bone pathology is common in Thoroughbred racehorses and believed to precede more serious injury. Early identification of pathology is critical to allow for intervention. To determine interobserver variability of fetlock subchondral bone lesions using cone beam and fan beam computed tomography (CBCT, FBCT) and to validate a robotics-controlled CBCT to identify fetlock subchondral bone pathology in the Thoroughbred racehorse. Prospective cohort study. FBCT and CBCT images were acquired of 25 metacarpo-/metatarsophalangeal joints of Thoroughbred racehorses. Images were analysed for subchondral bone lesions commonly identified in Thoroughbred fetlocks by an imaging specialist and surgery specialist. Interobserver and intermodality equivalence were determined with a Pearson correlation analysis and Bland-Altman equivalence test. Interobserver FBCT correlation was significant (P<.05) for 19 of 25 variables (Pearson R mean 0.77). Concordance was significant for all 25 variables (Bland Altman average difference 0.28±0.21mm). Interobserver CBCT correlation was significant for 21 of 25 variables (Pearson R mean 0.73). Concordance was significant for all variables (Bland Altman average difference 0.07±1.90mm). Intermodality (FBCT vs CBCT) correlation and concordance was significant for all variables as interpreted by the radiologist (Pearson R mean of 0.72, Bland Altman average difference 0.21±0.47mm). Intermodality correlation was significant for 19 of 25 variables as interpreted by the surgeon (Pearson R mean of 0.72). Concordance was significant for all variables (Bland Altman average difference 0.49±0.52mm). Neither FBCT nor CBCT images were compared with other imaging modalities/histopathology; limited number of cases included; inconsistent agreement of small lesions in specific categories. Standing CBCT is a valid diagnostic modality to identify subchondral bone lesions in Thoroughbred fetlocks. This technology may provide valuable information regarding the development and progression of fetlock pathology and yield insight into predisposing factors leading to more severe pathology.

Cross-modal subspace learning via kernel correlation maximization and discriminative structure-preserving

How to measure the distance between heterogeneous data is still an open problem. Many research works have been developed to learn a common subspace where the similarity between different modalities can be calculated directly. However, most of existing works focus on learning a latent subspace but the semantically structural information is not well preserved. Thus, these approaches cannot get desired results. In this paper, we propose a novel framework, termed Cross-modal subspace learning via Kernel correlation maximization and Discriminative structure-preserving (CKD), to solve this problem in two aspects. Firstly, we construct a shared semantic graph to make each modality data preserve the neighbor relationship semantically. Secondly, we introduce the Hilbert-Schmidt Independence Criteria (HSIC) to ensure the consistency between feature-similarity and semantic-similarity of samples. Our model not only considers the inter-modality correlation by maximizing the kernel correlation but also preserves the semantically structural information within each modality. The extensive experiments are performed to evaluate the proposed framework on the three public datasets. The experimental results demonstrate that the proposed CKD is competitive compared with the classic subspace learning methods.

Joint low rank embedded multiple features learning for audio–visual emotion recognition

Audio–visual emotion recognition is a challenging problem in the research fields of human–computer interaction and pattern recognition. Seeking a common subspace among the heterogeneous multi-modal data is essential for audio–visual emotion recognition. In this paper, we study the subspace learning for audio–visual emotion recognition by combing the similarity of intra-modality and the correlation of inter-modality. First, we enforce a low-rank constraint on the self-representation of the features in the subspace to exploit the structural similarity of intra-modality. It is based on a key observation that each modality and the corresponding features usually lie in a low-dimensional manifold. Second, we propose a joint low-rank model on the representation of inter-modality to keep consistency across different modalities. Finally, the intra-modality similarity and inter-modality correlation are integrated within a unified framework, for which we develop an efficient computational algorithm to pursue the common subspace. Experimental results on three typical audio–visual emotion datasets demonstrate the superior performance of our method on audio–visual emotion recognition.

Left atrial strain: a multi-modality, multi-vendor comparison study

Left atrial (LA) strain is a prognostic biomarker with utility across a spectrum of acute and chronic cardiovascular pathologies. There are limited data on intervendor differences and no data on intermodality differences for LA strain. We sought to compare the intervendor and intermodality differences between transthoracic echocardiography (TTE) and cardiac magnetic resonance (CMR) derived LA strain. We hypothesized that various components of atrial strain would show good intervendor and intermodality correlation but that there would be systematic differences between vendors and modalities. We evaluated 54 subjects (43 patients with a clinical indication for CMR and 11 healthy volunteers) in a study comparing TTE- and CMR-derived LA reservoir strain (ƐR), conduit strain (ƐCD), and contractile strain (ƐCT). The LA strain components were evaluated using four dedicated types of post-processing software. We evaluated the correlation and systematic bias between modalities and within each modality. Intervendor and intermodality correlation was: ƐR [intraclass correlation coefficient (ICC 0.64-0.90)], ƐCD (ICC 0.62-0.89), and ƐCT (ICC 0.58-0.77). There was evidence of systematic bias between vendors and modalities with mean differences ranging from (3.1-12.2%) for ƐR, ƐCD (1.6-8.6%), and ƐCT (0.3-3.6%). Reproducibility analysis revealed intraobserver coefficient of variance (COV) of 6.5-14.6% and interobserver COV of 9.9-18.7%. Vendor derived ƐR, ƐCD, and ƐCT demonstrates modest to excellent intervendor and intermodality correlation depending on strain component examined. There are systematic differences in measurements depending on modality and vendor. These differences may be addressed by future studies, which, examine calibration of LA geometry/higher frame rate imaging, semi-quantitative approaches, and improvements in reproducibility.

Left ventricular and atrial strain imaging with cardiac computed tomography: Validation against echocardiography

BackgroundData on left ventricular (LV) deformation imaging using CT angiography (CTA) are scarce and the feasibility of atrial deformation analysis by CT has not been addressed. We aimed to compare 2D echocardiographic and CT derived LV and left atrial (LA) global longitudinal strain (GLS) obtained by using a novel feature tracking algorithm in patients following transcatheter aortic valve implantation. MethodsTwenty-eight patients were included who underwent retrospectively-gated 256-slice CTA and speckle-tracking echocardiography (STE) on the same day. CT datasets in 10% increments were reconstructed throughout the cardiac cycle. LV GLS and LA global peak reservoir strain (LA GS) was measured. ResultsMedian absolute values for LV GLS were 19.9 [14.8–22.4] vs. 19.9 [16.8–24.7], as measured by CT vs STE, respectively (p = 0.017). We found good inter-modality correlation for LV GLS (ρ = 0.78, p < 0.05) with a mean bias of −1.6. Regarding atrial measurements, the median LA GS was 19.0 [13.5–27.3] for CT vs. 28.0 [17.5–32.6] for STE (p < 0.001) with a mean bias of −5.6 between CT and STE and a correlation coefficient of ρ = 0.87, p < 0.001. CT measurements were highly reproducible: intra-observer intra-class correlation coefficient was 0.96 for LV GLS and 0.95 for LA GS. ConclusionWe detected good correlation between CTA and echocardiography-based LV and LA longitudinal strain parameters. CTA provides accurate strain measurements with high reproducibility. Feature tracking-based deformation analysis could provide a clinically important addition to CT examinations by complementing anatomical information with functional data.

CM-GANs

It is known that the inconsistent distributions and representations of different modalities, such as image and text, cause the heterogeneity gap, which makes it very challenging to correlate heterogeneous data and measure their similarities. Recently, generative adversarial networks (GANs) have been proposed and have shown their strong ability to model data distribution and learn discriminative representation. It has also been shown that adversarial learning can be fully exploited to learn discriminative common representations for bridging the heterogeneity gap. Inspired by this, we aim to effectively correlate large-scale heterogeneous data of different modalities with the power of GANs to model cross-modal joint distribution. In this article, we propose Cross-modal Generative Adversarial Networks (CM-GANs) with the following contributions. First, a cross-modal GAN architecture is proposed to model joint distribution over the data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both compete with each other to promote cross-modal correlation learning. Second, the cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form the generative model. They not only exploit the cross-modal correlation for learning the common representations but also preserve reconstruction information for capturing the semantic consistency within each modality. Third, a cross-modal adversarial training mechanism is proposed, which uses two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make the generated common representations more discriminative by the adversarial training process. In summary, our proposed CM-GAN approach can use GANs to perform cross-modal common representation learning by which the heterogeneous data can be effectively correlated. Extensive experiments are conducted to verify the performance of CM-GANs on cross-modal retrieval compared with 13 state-of-the-art methods on 4 cross-modal datasets.