Multiple Images Research Articles

Computed Tomography Analysis of the Nasal Septal Deviation and Related Paranasal Sinus Pathologies

Purpose: Nasal Septum Deviation (NSD), which is defined as a dislocated septum, is a prevalent condition that is mostly asymptomatic; however, it can cause difficulties such as nasal obstruction and rhinosinusitis symptoms. Other Paranasal Sinuses (PNS) disorders, e.g., Osteo Meatal Complex (OMC) obstruction, have been assumed to be correlated with NSD. This study investigates the possible correlation of specific radiological findings of PNS pathologies in patients with or without the existence and direction of NSD. We also discussed the different types of NSD and their incidence among different groups stored by age, gender, etc. Materials and Methods: This is a cross-sectional study conducted at multiple imaging centers in Iran between July 2019 and February 2020 to estimate the prevalence of deviated nasal septum and correlate between NSD and other findings using paranasal sinus scans. All the individuals who met the inclusion and exclusion criteria were included in the study. NSD types have been classified based on Mladina's method. Results: We prospectively reviewed PNS Computed Tomography (CT) scans obtained from 254 cases (82 men and 172 women), aged 18 to 81 years (mean: 35.53), evaluated over seven months. Out of 254 cases, 135 (53 %) patients had NSD, without any direction predominance (right-sided vs. left-sided), and equally distributed between genders. The most common NSD types were type V with a frequency of 42.22% and type III with a frequency of 24.44%. Other PNS CT findings were also investigated in our patients, like OMC obstruction, pansinusitis, mucocele, air-fluid level, mucosal thickening, and concha bullosa. There was no statistically significant correlation between any of these pathologies and NSD (P-value > 0.05). Conclusion: Although NSD was observed in more than half of our cases' PNS CTs, there was no significant correlation between NSD and other findings in most subjects. This reveals that the effectiveness of septal repair surgeries in relieving patients' symptoms might not be as high as expected.

Weighted Contrastive Prototype Network for Few-Shot Hyperspectral Image Classification with Noisy Labels

Few-shot hyperspectral image classification aims to develop the ability of classifying image pixels by using relatively few labeled pixels per class. However, due to the inaccuracy of the localization system and the bias of the ground survey, the potential noisy labels in the training data pose a very significant challenge to few-shot hyperspectral image classification. To solve this problem, this paper proposes a weighted contrastive prototype network (WCPN) for few-shot hyperspectral image classification with noisy labels. WCPN first utilizes a similarity metric to generate the weights of the samples from the same classes, and applies them to calibrate the class prototypes of support and query sets. Then the weighted prototype network will minimize the distance between features and prototypes to train the network. WCPN also incorporates a weighted contrastive regularization function that uses the sample weights as gates to filter the fake positive samples whose labels are incorrect to further improve the discriminative power of the prototypes. We conduct experiments on multiple hyperspectral image datasets with artificially generated noisy labels, and the results show that the WCPN has excellent performance that can sufficiently mitigate the impact of noisy labels.

Statistical analysis of multiple regions-of-interest in multiplexed spatial proteomics data.

Multiplexed spatial proteomics reveals the spatial organization of cells in tumors, which is associated with important clinical outcomes such as survival and treatment response. This spatial organization is often summarized using spatial summary statistics, including Ripley's K and Besag's L. However, if multiple regions of the same tumor are imaged, it is unclear how to synthesize the relationship with a single patient-level endpoint. We evaluate extant approaches for accommodating multiple images within the context of associating summary statistics with outcomes. First, we consider averaging-based approaches wherein multiple summaries for a single sample are combined in a weighted mean. We then propose a novel class of ensemble testing approaches in which we simulate random weights used to aggregate summaries, test for an association with outcomes, and combine the $P$-values. We systematically evaluate the performance of these approaches via simulation and application to data from non-small cell lung cancer, colorectal cancer, and triple negative breast cancer. We find that the optimal strategy varies, but a simple weighted average of the summary statistics based on the number of cells in each image often offers the highest power and controls type I error effectively. When the size of the imaged regions varies, incorporating this variation into the weighted aggregation may yield additional power in cases where the varying size is informative. Ensemble testing (but not resampling) offered high power and type I error control across conditions in our simulated data sets.

Single‐shot multi‐b‐value (SSMb) diffusion‐weighted MRI using spin echo and stimulated echoes with variable flip angles

AbstractConventional diffusion‐weighted imaging (DWI) sequences employing a spin echo or stimulated echo sensitize diffusion with a specific b‐value at a fixed diffusion direction and diffusion time (Δ). To compute apparent diffusion coefficient (ADC) and other diffusion parameters, the sequence needs to be repeated multiple times by varying the b‐value and/or gradient direction. In this study, we developed a single‐shot multi‐b‐value (SSMb) diffusion MRI technique, which combines a spin echo and a train of stimulated echoes produced with variable flip angles. The method involves a pair of 90° radio frequency (RF) pulses that straddle a diffusion gradient lobe (GD), to rephase the magnetization in the transverse plane, producing a diffusion‐weighted spin echo acquired by the first echo‐planar imaging (EPI) readout train. The magnetization stored along the longitudinal axis is successively re‐excited by a series of n variable‐flip‐angle pulses, each followed by a diffusion gradient lobe GD and a subsequent EPI readout train to sample n stimulated‐echo signals. As such, (n + 1) diffusion‐weighted images, each with a distinct b‐value, are acquired in a single shot. The SSMb sequence was demonstrated on a diffusion phantom and healthy human brain to produce diffusion‐weighted images, which were quantitative analyzed using a mono‐exponential model. In the phantom experiment, SSMb provided similar ADC values to those from a commercial spin‐echo EPI (SE‐EPI) sequence (r = 0.999). In the human brain experiment, SSMb enabled a fourfold scan time reduction and yielded slightly lower ADC values (0.83 ± 0.26 μm2/ms) than SE‐EPI (0.88 ± 0.29 μm2/ms) in all voxels excluding cerebrospinal fluid, likely due to the influence of varying diffusion times. The feasibility of using SSMb to acquire multiple images in a single shot for intravoxel incoherent motion (IVIM) analysis was also demonstrated. In conclusion, despite a relatively low signal‐to‐noise ratio, the proposed SSMb technique can substantially increase the data acquisition efficiency in DWI studies.

A General On-Orbit Absolute Radiometric Calibration Method Compatible with Multiple Imaging Conditions

On-orbit absolute radiometric calibration is not only a prerequisite for the quantitative application of optical remote sensing satellite data but also a key step in ensuring the accuracy and reliability of satellite observation data. Due to the diversity of imaging conditions for optical remote sensing satellite sensors, on-orbit absolute radiometric calibration usually requires a large number of imaging tasks and manual labor to calibrate each imaging condition. This seriously limits the timeliness of on-orbit absolute radiometric calibration and is also an urgent problem to be solved in the context of the explosive growth of satellite numbers. Based on this, we propose a general on-orbit absolute radiometric calibration method compatible with multiple imaging conditions. Firstly, we use a large amount of laboratory radiometric calibration data to explore the mathematical relationship between imaging conditions (row transfer time, integration level and gain), radiance, and DN, and successfully build an imaging condition compatibility model. Secondly, we combine the imaging condition compatibility model with cross calibration to achieve a general on-orbit absolute radiometric calibration method. We use cross calibration to obtain the reference radiance and corresponding DN of the target satellites, which calculates the general coefficient by using row transfer time, integration level, and gain, and use the general coefficient to calibrate all imaging conditions. Finally, we use multiple imaging tasks of the JL1GF03D11 satellites to verify the effectiveness of the proposed method. The experiments show that the average relative difference was reduced to 2.79% and the RMSE was reduced to 1.51, compared with the laboratory radiometric calibration method. In addition, we also verify the generality of the proposed method by using 10 satellites of the Jilin-1 GF03D series. The experiment shows that the goodness of fit of the general coefficient is all greater than 95%, and the average relative difference between the reference radiance and the calibrated radiance of the proposed method is 2.46%, with an RMSE of 1.67. To sum up, by using the proposed method, all imaging conditions of optical remote sensing satellite sensor can be calibrated in one imaging task, which greatly improves the timeliness and accuracy of on-orbit absolute radiometric calibration.

Feature-Selection-Based Unsupervised Transfer Learning for Change Detection from VHR Optical Images

Accurate understanding of urban land use change information is of great significance for urban planning, urban monitoring, and disaster assessment. The use of Very-High-Resolution (VHR) remote sensing images for change detection on urban land features has gradually become mainstream. However, most existing transfer learning-based change detection models compute multiple deep image features, leading to feature redundancy. Therefore, we propose a Transfer Learning Change Detection Model Based on Change Feature Selection (TL-FS). The proposed method involves using a pretrained transfer learning model framework to compute deep features from multitemporal remote sensing images. A change feature selection algorithm is then designed to filter relevant change information. Subsequently, these change features are combined into a vector. The Change Vector Analysis (CVA) is employed to calculate the magnitude of change in the vector. Finally, the Fuzzy C-Means (FCM) classification is utilized to obtain binary change detection results. In this study, we selected four VHR optical image datasets from Beijing-2 for the experiment. Compared with the Change Vector Analysis and Spectral Gradient Difference, the TL-FS method had maximum increases of 26.41% in the F1-score, 38.04% in precision, 29.88% in recall, and 26.15% in the overall accuracy. The results of the ablation experiments also indicate that TL-FS could provide clearer texture and shape detections for dual-temporal VHR image changes. It can effectively detect complex features in urban scenes.

Accuracy assessment of UAV-based documentation of archaeological site: Kültepe-Kaneš

High quality 2D/3D metric documentation of archaeological sites is used in many different applications such as archaeological-heritage management, digital archaeological excavation planning, orthophotographic documentation, and ancient Digital Surfaces Models (DSM) research. Structure-from-Motion based photogrammetry (SfM) is an indirect geodesic measurement technique based on multiple view image processing using high-level automation techniques. Using SfM to generate the high accuracy 2D/3D data needed for digital documentation of archaeological cultural heritage is relatively practical, economical, and fast. Factors such as the number of ground control points and their distribution to the study area, the robustness of the camera system model parameters, and the analytical model of the photogrammetric triangulation method used affect the accuracy of the data produced using SfM-photogrammetry. Kültepe-Kaneš, the center of the ancient Kingdom of Kaneš, which is located in central-Anatolia, was an Assyrian Trade Colony active between ca. 2000 BC and 1700 BC. The site, which was the center of the Old Assyrian trade colonies in Anatolia and one of the most important sites in ancient history, has been under excavation for the last 75 years. In this paper, high accuracy digital documentation of the Kültepe-Kaneš archaeological excavation area was made using different SfM methods. Statistical analysis revealed no significant difference in accuracy between Real-Time Kinematic (RTK) and Post-Process Kinematic (PPK) methods for the study area. This suggests that either PPK, RTK, or aerial triangulation with Ground Control Points (GCPs) can be employed to generate orthophotos and DSMs for archaeological sites with relatively flat surface areas, as long as a sufficient number of uniformly distributed GCPs are used.

Triple-Camera Rectification for Depth Estimation Sensor.

In this study, we propose a novel rectification method for three cameras using a single image for depth estimation. Stereo rectification serves as a fundamental preprocessing step for disparity estimation in stereoscopic cameras. However, off-the-shelf depth cameras often include an additional RGB camera for creating 3D point clouds. Existing rectification methods only align two cameras, necessitating an additional rectification and remapping process to align the third camera. Moreover, these methods require multiple reference checkerboard images for calibration and aim to minimize alignment errors, but often result in rotated images when there is significant misalignment between two cameras. In contrast, the proposed method simultaneously rectifies three cameras in a single shot without unnecessary rotation. To achieve this, we designed a lab environment with checkerboard settings and obtained multiple sample images from the cameras. The optimization function, designed specifically for rectification in stereo matching, enables the simultaneous alignment of all three cameras while ensuring performance comparable to traditional methods. Experimental results with real camera samples demonstrate the benefits of the proposed method and provide a detailed analysis of unnecessary rotations in the rectified images.

Improved multi-focus image fusion using online convolutional sparse coding based on sample-dependent dictionary

Multi-focus image fusion merges multiple images captured from different focused regions of a scene to create a fully-focused image. Convolutional sparse coding (CSC) methods are commonly employed for accurate extraction of focused regions, but they often disregard computational costs. To overcome this, an online convolutional sparse coding (OCSC) technique was introduced, but its performance is still limited by the number of filters used, affecting overall performance negatively. To address these limitations, a novel approach called Sample-Dependent Dictionary-based Online Convolutional Sparse Coding (SCSC) was proposed. SCSC enables the utilization of additional filters while maintaining low time and space complexity for processing high-dimensional or large data. Leveraging the computational efficiency and effective global feature extraction of SCSC, we propose a novel method for multi-focus image fusion. Our method involves a two-layer decomposition of each source image, yielding a base layer capturing the predominant features and a detail layer containing finer details. The amalgamation of the fused base and detail layers culminates in the reconstruction of the final image. The proposed method significantly mitigates artifacts, preserves fine details at the focus boundary, and demonstrates notable enhancements in both visual quality and objective evaluation of multi-focus image fusion.

Gastrointestinal image stitching based on improved unsupervised algorithm.

Image stitching is a traditional but challenging computer vision task. The goal is to stitch together multiple images with overlapping areas into a single, natural-looking, high-resolution image without ghosts or seams. This article aims to increase the field of view of gastroenteroscopy and reduce the missed detection rate. To this end, an improved depth framework based on unsupervised panoramic image stitching of the gastrointestinal tract is proposed. In addition, preprocessing for aberration correction of monocular endoscope images is introduced, and a C2f module is added to the image reconstruction network to improve the network's ability to extract features. A comprehensive real image data set, GASE-Dataset, is proposed to establish an evaluation benchmark and training learning framework for unsupervised deep gastrointestinal image splicing. Experimental results show that the MSE, RMSE, PSNR, SSIM and RMSE_SW indicators are improved, while the splicing time remains within an acceptable range. Compared with traditional image stitching methods, the performance of this method is enhanced. In addition, improvements are proposed to address the problems of lack of annotated data, insufficient generalization ability and insufficient comprehensive performance in image stitching schemes based on supervised learning. These improvements provide valuable aids in gastrointestinal examination.

Self-Supervised 3D Semantic Occupancy Prediction from Multi-View 2D Surround Images

An accurate 3D representation of the geometry and semantics of an environment builds the basis for a large variety of downstream tasks and is essential for autonomous driving related tasks such as path planning and obstacle avoidance. The focus of this work is put on 3D semantic occupancy prediction, i.e., the reconstruction of a scene as a voxel grid where each voxel is assigned both an occupancy and a semantic label. We present a Convolutional Neural Network-based method that utilizes multiple color images from a surround-view setup with minimal overlap, together with the associated interior and exterior camera parameters as input, to reconstruct the observed environment as a 3D semantic occupancy map. To account for the ill-posed nature of reconstructing a 3D representation from monocular 2D images, the image information is integrated over time: Under the assumption that the camera setup is moving, images from consecutive time steps are used to form a multi-view stereo setup. In exhaustive experiments, we investigate the challenges presented by dynamic objects and the possibilities of training the proposed method with either 3D or 2D reference data. Latter being motivated by the comparably higher costs of generating and annotating 3D ground truth data. Moreover, we present and investigate a novel self-supervised training scheme that does not require any geometric reference data, but only relies on sparse semantic ground truth. An evaluation on the Occ3D dataset, including a comparison against current state-of-the-art self-supervised methods from the literature, demonstrates the potential of our self-supervised variant.

Multidimensional data visualization and synchronization for revealing hidden pandemic information

Visualization is integral to uncovering hidden information in data and providing users with intuitive feedback for decision-making. Data visualization is crucial for transforming complex data into actionable insights across various domains. In recent years, coronavirus disease vaccines have become increasingly available to much of the population. However, the CDC (Centers for Disease Control and Prevention) often fails to consider multidimensional coronavirus pandemic data from a side-by-side perspective, limiting the ability of medical professionals and individuals to compare and interact with comprehensive data visualizations. Effectively displaying coronavirus and vaccination data collected from multiple sources is essential for interpreting pandemic transmission patterns and vaccine efficiency. This paper presents a new platform for innovative data visualizations that offers users intuitive feedback and a complete data story. We designed algorithms to seamlessly combine multiple parameters, synchronize attributes, and dynamically visualize data over time on a single webpage. Instead of integrating all attributes into a single plot, which can be overwhelming due to space limitations and make it difficult to extract crucial information from overcrowded display components, we developed algorithms to classify, enhance, and group all parameters based on their relationships and similarities. Furthermore, a side-by-side visualization method was created to dynamically link all parameters in multiple images for data exploration, trend comparison, hidden information detection, and correspondence analysis. Our platform provides real-time performance, enabling healthcare professionals to make informed decisions, communicate findings effectively, and uncover patterns that might not be apparent in raw data. The proposed multidimensional data visualization algorithms have broad applications in general data exploration and revealing hidden information.

MSI-DTrans: A multi-focus image fusion using multilayer semantic interaction and dynamic transformer

Multi-focus image fusion (MFIF) aims to utilize multiple images with different focal lengths to fuse into a single full-focus image. This process enhances the realism and clarity of the resulting image. In this paper, a MFIF method called MSI-DTrans was proposed. On the one hand, in order to fully utilize all the effective information that the source image carries, the proposed method adopts a multilayer semantic interaction strategy to enhance the interaction of high-frequency and low-frequency information. This approach gradually mines more abstract semantic information, guiding the generation of feature maps from coarse to fine. On the other hand, a parallel multi-scale joint self-attention computation model is designed. The model adopts dynamic sense field and dynamic token embedding to overcome the performance degradation problem when dealing with multi-scale objects. This enables self-attention to integrate long-range dependencies between objects of different scales and reduces computational overhead. Numerous experimental results show that the proposed method effectively avoids image distortion, achieves better visualization results, and demonstrates good competitiveness with many state-of-the-art methods in terms of qualitative and quantitative analysis, as well as efficiency comparison. The source code is available at https://github.com/ouyangbaicai/MSI-DTrans.

Diverse Face Images (DFI): Validated for racial representation and eye gaze.

Face processing is a central component of human communication and social engagement. The present investigation introduces a set of racially and ethnically inclusive faces created for researchers interested in perceptual and socio-cognitive processes linked to human faces. The Diverse Face Images (DFI) stimulus set includes high-quality still images of female faces that are racially and ethnically representative, include multiple images of direct and indirect gaze for each model and control for low-level perceptual variance between images. The DFI stimuli will support researchers interested in studying face processing throughout the lifespan as well as other questions that require a diversity of faces or gazes. This report includes a detailed description of stimuli development and norming data for each model. Adults completed a questionnaire rating each image in the DFI stimuli set on three major qualities relevant to face processing: (1) strength of race/ethnicity group associations, (2) strength of eye gaze orientation, and (3) strength of emotion expression. These validation data highlight the presence of rater variability within and between individual model images as well as within and between race and ethnicity groups.

Fast readout of identity-related information in multiple images after incidental learning

Fast readout of identity-related information in multiple images after incidental learning

Predictive value of mono-exponential and multiple mathematical models in locally advanced rectal cancer response to neoadjuvant chemoradiotherapy.

This prospective study aimed to assess the predictive value of mono-exponential and multiple mathematical diffusion-weighted imaging (DWI) models in determining the response to neoadjuvant chemoradiotherapy (nCRT) in patients with locally advanced rectal cancer (LARC). The study included 103 LARC patients scheduled for preoperative chemoradiotherapy between December 2021 and June 2023 Magnetic resonance imaging (MRI) scans were performed using a 3.0-T MR scanner, encompassing sagittal, axial, and oblique coronal T2-weighted images without fat saturation, along with DWI perpendicular to the rectum's long axis. Various DWI parameters, including apparent diffusion coefficient (ADC), stretched exponential model (SEM), continuous-time random-walk model (CTRW), and fractional-order calculus model (FROC), were measured. The pathologic complete response (pCR) rate and tumor downstaging (T-downstage) rate were determined. After nCRT, SEM-α, SEM-DDC, CTRW-α, CTRW-β, CTRW-D, FROC-β, and ADC values were significantly higher in the pCR group compared to the non-pCR group (all P < 0.05). SEM-DDC, CTRW-α, CTRW-D, FROC-β, FROC-µ, and ADC values were significantly higher in the T-downstage group (ypT0-1) than in the non-T-downstage group (ypT2-4) (P < 0.05). The combination of CTRW (α + β + D) exhibited the best diagnostic performance for assessing pCR after nCRT (AUC = 0.840, P < 0.001). Pre-nCRT CTRW (α + β) demonstrated a predictive AUC of 0.652 (95%CI: 0.552-0.743), 90.3% sensitivity, and 43.1% specificity for pCR. Regarding T-downstage assessment after nCRT, the combination of CTRW (α + D) yielded the best diagnostic performance (AUC = 0.877, P = 0.048). In LARC patients, imaging markers derived from CTRW show promise in predicting tumor response before nCRT and assessing pCR after nCRT.

A case of chronic contained rupture of abdominal aortic aneurysm mimicking a retroperitoneal tumor.

Chronic contained rupture of abdominal aortic aneurysm (CCR-AAA) is a rare subtype of abdominal aortic rupturethat can mimic other retroperitoneal lesions. We report a case of CCR-AAA in a man in his sixties who presented witha 10-month history of right low back pain and weight loss. Contrast-enhanced computed tomography (CT) revealed alobulated retroperitoneal mass around the abdominal aorta, initially misdiagnosed as a possible hemorrhagicretroperitoneal tumor. Despite multiple imaging studies including CT, magnetic resonance imaging (MRI), and 18F-fluorodeoxyglucosepositron emission tomography (FDG-PET)/CT, as well as a CT-guided biopsy, the correct diagnosisremained elusive for 34 months. Key findings included subtle aortic wall irregularity on CT, high signal intensity onT1-weighted MRI suggesting hematoma, peripheral FDG uptake on PET/CT, and histological findings of biopsy tissueconsistent with organizing hematoma. Surgery confirmed the diagnosis, revealing an organized hematoma with adefect in the right wall of the abdominal aortic aneurysm. This case demonstrates that CCR-AAA can present withatypical radiological features, potentially leading to misdiagnosis. When encountering a para-aortic mass with ahemorrhagic component, careful observation of the AAA morphology and aortic wall contour is crucial for anaccurate diagnosis of CCR-AAA.

Long-term mesoscale imaging of 3D intercellular dynamics across a mammalian organ

A comprehensive understanding of physio-pathological processes necessitates non-invasive intravital three-dimensional (3D) imaging over varying spatial and temporal scales. However, huge data throughput, optical heterogeneity, surface irregularity, and phototoxicity pose great challenges, leading to an inevitable trade-off between volume size, resolution, speed, sample health, and system complexity. Here, we introduce a compact real-time, ultra-large-scale, high-resolution 3D mesoscope (RUSH3D), achieving uniform resolutions of 2.6×2.6×6μm3 across a volume of 8,000×6,000×400μm3 at 20Hz with low phototoxicity. Through the integration of multiple computational imaging techniques, RUSH3D facilitates a 13-fold improvement in data throughput and an orders-of-magnitude reduction in system size and cost. With these advantages, we observed premovement neural activity and cross-day visual representational drift across the mouse cortex, the formation and progression of multiple germinal centers in mouse inguinal lymph nodes, and heterogeneous immune responses following traumatic brain injury-all at single-cell resolution, opening up a horizon for intravital mesoscale study of large-scale intercellular interactions at the organ level.

Accelerated model-based T1, T2* and proton density mapping using a Bayesian approach with automatic hyperparameter estimation.

To achieve automatic hyperparameter estimation for the model-based recovery of quantitative MR maps from undersampled data, we propose a Bayesian formulation that incorporates the signal model and sparse priors among multiple image contrasts. We introduce a novel approximate message passing framework "AMP-PE" that enables the automatic and simultaneous recovery of hyperparameters and quantitative maps. We employed the variable-flip-angle method to acquire multi-echo measurements using gradient echo sequence. We explored undersampling schemes to incorporate complementary sampling patterns across different flip angles and echo times. We further compared AMP-PE with conventional compressed sensing approaches such as the -norm minimization, PICS and other model-based approaches such as GraSP, MOBA. Compared to conventional compressed sensing approaches such as the -norm minimization and PICS, AMP-PE achieved superior reconstruction performance with lower errors in mapping and comparable performance in and proton density mappings. When compared to other model-based approaches including GraSP and MOBA, AMP-PE exhibited greater robustness and outperformed GraSP in reconstruction error. AMP-PE offers faster speed than MOBA. AMP-PE performed better than MOBA at higher sampling rates and worse than MOBA at a lower sampling rate. Notably, AMP-PE eliminates the need for hyperparameter tuning, which is a requisite for all the other approaches. AMP-PE offers the benefits of model-based recovery with the additional key advantage of automatic hyperparameter estimation. It works adeptly in situations where ground-truth is difficult to obtain and in clinical environments where it is desirable to automatically adapt hyperparameters to individual protocol, scanner and patient.

An hetero-modal deep learning framework for medical image synthesis applied to contrast and non-contrast MRI

Some pathologies such as cancer and dementia require multiple imaging modalities to fully diagnose and assess the extent of the disease. Magnetic resonance imaging offers this kind of polyvalence, but examinations take time and can require contrast agent injection. The flexible synthesis of these imaging sequences based on the available ones for a given patient could help reduce scan times or circumvent the need for contrast agent injection. In this work, we propose a deep learning architecture that can perform the synthesis of all missing imaging sequences from any subset of available images. The network is trained adversarially, with the generator consisting of parallel 3D U-Net encoders and decoders that optimally combines their multi-resolution representations with a fusion operation learned by an attention network trained conjointly with the generator network. We compare our synthesis performance with 3D networks using other types of fusion and a comparable number of trainable parameters, such as the mean/variance fusion. In all synthesis scenarios except one, the synthesis performance of the network using attention-guided fusion was better than the other fusion schemes. We also inspect the encoded representations and the attention network outputs to gain insights into the synthesis process, and uncover desirable behaviors such as prioritization of specific modalities, flexible construction of the representation when important modalities are missing, and modalities being selected in regions where they carry sequence-specific information. This work suggests that a better construction of the latent representation space in hetero-modal networks can be achieved by using an attention network.