Super-resolution Process Research Articles

Blind super-resolution network based on local fuzzy discriminative loss for fabric data augmentation

In the field of fabric defect detection, the development of algorithms has been hindered by issues such as poor quality and limited quantity of open-source datasets. Traditional data augmentation methods offer limited improvements in model performance, while generative data augmentation methods are plagued by difficulties in training generative models, susceptibility to artifacts, and the need for re-labeling. To address these challenges, this paper proposes a blind super-resolution algorithm for fabric defect data augmentation. The model is based on Real-ESRGAN and has been optimized specifically for the resolution degradation module to better adapt to the resolution degradation process in fabric images. Subsequently, a novel loss function named Local Blur Discrimination Loss is designed to address the local blur phenomenon and suppress the generation of fabric artifacts during the super-resolution process. Finally, both subjective evaluations of super-resolution effects and objective comparisons of data augmentation performance were conducted during the experimental phase. The subjective assessments demonstrate that the proposed method outperforms the baseline model. Additionally, in terms of objective performance, augmenting the DAGM2007 dataset using the proposed model, the detection model's accuracy (P) increased by 7.4%, recall (R) increased by 1.0%, and the mean average precision (mAP) increased by 2.5%, surpassing commonly used traditional vision-based data augmentation algorithms.

Dataset augmentation with multiple contrasts images in super-resolution processing of T1-weighted brain magnetic resonance images.

This study investigated the effectiveness of augmenting datasets for super-resolution processing of brain Magnetic Resonance Images (MRI) T1-weighted images (T1WIs) using deep learning. By incorporating images with different contrasts from the same subject, this study sought to improve network performance and assess its impact on image quality metrics, such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). This retrospective study included 240 patients who underwent brain MRI. Two types of datasets were created: the Pure-Dataset group comprising T1WIs and the Mixed-Dataset group comprising T1WIs, T2-weighted images, and fluid-attenuated inversion recovery images. A U-Net-based network and an Enhanced Deep Super-Resolution network (EDSR) were trained on these datasets. Objective image quality analysis was performed using PSNR and SSIM. Statistical analyses, including paired t test and Pearson's correlation coefficient, were conducted to evaluate the results. Augmenting datasets with images of different contrasts significantly improved training accuracy as the dataset size increased. PSNR values ranged 29.84-30.26dB for U-Net trained on mixed datasets, and SSIM values ranged 0.9858-0.9868. Similarly, PSNR values ranged 32.34-32.64dB for EDSR trained on mixed datasets, and SSIM values ranged 0.9941-0.9945. Significant differences in PSNR and SSIM were observed between models trained on pure and mixed datasets. Pearson's correlation coefficient indicated a strong positive correlation between dataset size and image quality metrics. Using diverse image data obtained from the same subject can improve the performance of deep-learning models in medical image super-resolution tasks.

Super-Resolution Processing for Multiple Aperture Antenna to Suppress Multipath

Angle estimation for low-altitude targets above the sea surface is a challenging problem due to multipath interference from surface reflection signals, and various approaches have been proposed. This paper proposes a matrix pencil method with multiple apertures. The matrix pencil method effectively responds to dynamic scenarios because it performs better when using a single snapshot than other methods. Also, employing multiple apertures is more economical than using one large aperture. Therefore, we propose a computationally efficient approach using this method and structures. The proposed two-stage MP method incrementally improves the resolution in two stages: in stage 1, we extract the denoised signals at each aperture level, and in stage 2, we further improve the resolution with those signals. In comparison with the angular resolution defined by the half-power beamwidth (HPBW) of a uniform linear array (ULA) antenna with an equivalent number of arrays, the proposed method demonstrated a superior resolution of less than 0.087 of the HPBW at a high signal-to-noise ratio (SNR) of 40 dB, and less than 0.31 of it even at a relatively low SNR of 15 dB, based on 90% of the resolving probability. For the multipath problem, the proposed scheme has the advantage of not requiring prior geometric information, and its performance is demonstrated through simulations to be better than the adaptive beamforming method and the composite monopulse method.

Generative AI-Enabled Energy-Efficient Mobile Augmented Reality in Multi-Access Edge Computing

This paper proposes a novel offloading and super-resolution (SR) control scheme for energy-efficient mobile augmented reality (MAR) in multi-access edge computing (MEC) using SR as a promising generative artificial intelligence (GAI) technology. Specifically, SR can enhance low-resolution images into high-resolution versions using GAI technologies. This capability is particularly advantageous in MAR by lowering the bitrate required for network transmission. However, this SR process requires considerable computational resources and can introduce latency, potentially overloading the MEC server if there are numerous offload requests for MAR services. In this context, we conduct an empirical study to verify that the computational latency of SR increases with the upscaling level. Therefore, we demonstrate a trade-off between computational latency and improved service satisfaction when upscaling images for object detection, as it enhances the detection accuracy. From this perspective, determining whether to apply SR for MAR, while jointly controlling offloading decisions, is challenging. Consequently, to design energy-efficient MAR, we rigorously formulate analytical models for the energy consumption of a MAR device, the overall latency and the MAR satisfaction of service quality from the enforcement of the service accuracy, taking into account the SR process at the MEC server. Finally, we develop a theoretical framework that optimizes the computation offloading and SR control problem for MAR clients by jointly optimizing the offloading and SR decisions, considering their trade-off in MAR with MEC. Finally, the performance evaluation indicates that our proposed framework effectively supports MAR services by efficiently managing offloading and SR decisions, balancing trade-offs between energy consumption, latency, and service satisfaction compared to benchmarks.

A Super-Resolution and 3D Reconstruction Method Based on OmDF Endoscopic Images.

In the field of endoscopic imaging, challenges such as low resolution, complex textures, and blurred edges often degrade the quality of 3D reconstructed models. To address these issues, this study introduces an innovative endoscopic image super-resolution and 3D reconstruction technique named Omni-Directional Focus and Scale Resolution (OmDF-SR). This method integrates an Omnidirectional Self-Attention (OSA) mechanism, an Omnidirectional Scale Aggregation Group (OSAG), a Dual-stream Adaptive Focus Mechanism (DAFM), and a Dynamic Edge Adjustment Framework (DEAF) to enhance the accuracy and efficiency of super-resolution processing. Additionally, it employs Structure from Motion (SfM) and Multi-View Stereo (MVS) technologies to achieve high-precision medical 3D models. Experimental results indicate significant improvements in image processing with a PSNR of 38.2902 dB and an SSIM of 0.9746 at a magnification factor of ×2, and a PSNR of 32.1723 dB and an SSIM of 0.9489 at ×4. Furthermore, the method excels in reconstructing detailed 3D models, enhancing point cloud density, mesh quality, and texture mapping richness, thus providing substantial support for clinical diagnosis and surgical planning.

High-flexibility reconstruction of small-scale motions in wall turbulence using a generalized zero-shot learning

This study proposes a novel super-resolution (or SR) framework for generating high-resolution turbulent boundary layer (TBL) flow from low-resolution inputs. The framework combines a super-resolution generative adversarial neural network (SRGAN) with down-sampling modules (DMs), integrating the residual of the continuity equation into the loss function. The DMs selectively filter out components with excessive energy dissipation in low-resolution fields prior to the super-resolution process. The framework iteratively applies the SRGAN and DM procedure to fully capture the energy cascade of multi-scale flow structures, collectively termed the SRGAN-based energy cascade reconstruction framework (EC-SRGAN). Despite being trained solely on turbulent channel flow data (via ‘zero-shot transfer’), EC-SRGAN exhibits remarkable generalization in predicting TBL small-scale velocity fields, accurately reproducing wavenumber spectra compared to direct numerical simulation (DNS) results. Furthermore, a super-resolution core is trained at a specific super-resolution ratio. By leveraging this pretrained super-resolution core, EC-SRGAN efficiently reconstructs TBL fields at multiple super-resolution ratios from various levels of low-resolution inputs, showcasing strong flexibility. By learning turbulent scale invariance, EC-SRGAN demonstrates robustness across different TBL datasets. These results underscore the potential of EC-SRGAN for generating and predicting wall turbulence with high flexibility, offering promising applications in addressing diverse TBL-related challenges.

Research into super-resolution in medical imaging from 2000 to 2023: bibliometric analysis and visualization.

Super-resolution (SR) refers to the use of hardware or software methods to enhance the resolution of low-resolution (LR) images and produce high-resolution (HR) images. SR is applied frequently across a variety of medical imaging contexts, particularly in the enhancement of neuroimaging, with specific techniques including SR microscopy-used for diagnostic biomarkers-and functional magnetic resonance imaging (fMRI)-a neuroimaging method for the measurement and mapping of brain activity. This bibliometric analysis of the literature related to SR in medical imaging was conducted to identify the global trends in this field, and visualization via graphs was completed to offer insights into future research prospects. In order to perform a bibliometric analysis of the SR literature, this study sourced all publications from the Web of Science Core Collection (WoSCC) database published from January 1, 2000, to October 11, 2023. A total of 3,262 articles on SR in medical imaging were evaluated. VOSviewer was used to perform co-occurrence and co-authorship analysis, and network visualization of the literature data, including author, journal, publication year, institution, and keywords, was completed. From 2000 to 2023, the annual publication volume surged from 13 to 366. The top three journals in this field in terms of publication volume were as follows: (I) Scientific Reports (86 publications), (II) IEEE Transactions on Medical Imaging (74 publications), and (III) IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control (56 publications). The most prolific country, institution, and author were the United States (1,017 publications; 31,301 citations), the Chinese Academy of Sciences (124 publications; 2,758 citations), and Dinggang Shen (20 publications; 671 citations), respectively. A cluster analysis of the top 100 keywords was conducted, which revealed the presence of five co-occurrence clusters: (I) SR and artificial intelligence (AI) for medical image enhancement, (II) SR and inverse problem processing concepts for positron emission tomography (PET) image processing, (III) SR ultrasound through microbubbles, (IV) SR microscopy for Alzheimer and Parkinson diseases, and (V) SR in brain fMRI: rapid acquisition and precise imaging. The most recent high-frequency keywords were deep learning (DL), magnetic resonance imaging (MRI), and convolutional neural networks (CNNs). Over the past two decades, the output of publications by countries, institutions, and authors in the field of SR in medical imaging has steadily increased. Based on bibliometric analysis of international trends, the resurgence of SR in medical imaging has been facilitated by advancements in AI. The increasing need for multi-center and multi-modal medical images has further incentivized global collaboration, leading to the diverse research paths in SR medical imaging among prominent scientists.

Enhancing Inductive IR Thermography by Using FFT-Equalization, Motion Tracking Detection and VDSR Super-resolution Processing

Among non-destructive inspection techniques, infrared thermography stands out as a promising technology that enables real-time inspection of large areas without the need for physical contact. In this study, we employed the dynamic induction thermography method, which is one of the active infrared thermography techniques, to detect defects on the back side of the S275 material specimen. This technique involves creating relative movement between the IR camera and the specimen. We acquired sequence images at different moving speeds using the induction thermography technique, and then used the FFT with Gaussian filtering to solve for non-uniform heat sources. To further enhance the resolution, we applied the VDSR technique, which is based on deep neural networks. The effectiveness of this approach was validated both qualitatively and quantitatively. Finally, we utilized the MOT algorithm to automatically detect defects in the image with the highest thermal contrast, which was captured at a speed of 15 mm/s. In this study, we demonstrate the effectiveness of thermal equalization using the GF-based FFT algorithm, as well as the superresolution conversion achieved through the VDSR-based deep neural network. Additionally, we present a mechanism for automated slit detection using the MOT algorithm.

Light field image super-resolution using a Content-Aware Spatial–Angular Interaction network

Light field images record a series of viewpoints of a scene and thus have many attractive applications. However, the trade-off between angular and spatial resolution in the imaging process makes light field image super-resolution necessary. In this paper, we propose a Content-Aware Spatial–Angular Interaction (dubbed CASAI) network for light field image super-resolution. The gradient branch of CASAI makes full use of the context information of the low-resolution gradient map and the multi-level features of the super-resolution branch to generate the high-resolution gradient map, which enables the awareness of structure, texture and detail, and provides effective prior knowledge for the super-resolution process. The super-resolution branch of CASAI generates high-quality super-resolution images by making full use of intra-view (i.e., spatial) and inter-view (i.e., angular) information through a spatial–angular adaptive interaction block and using the high-resolution gradient prior as guidance. The spatial–angular adaptive interaction block enables the awareness of the different importance of spatial features and angular features, so as to better integrate intra-view and inter-view information to improve the performance of light field image super-resolution. The experimental results indicate that when the upsampling factor is 4, our method outperforms our baseline (LF-InterNet) with an average PSNR increase of 0.48 dB, while also achieving the best SSIM. Visualization results demonstrate the advantage of our method in simultaneously generating natural SR images and restoring structures.

Research on image restoration of [formula omitted] porous foam ceramics media using Transformer architecture

Pore structure images of foam ceramics play a crucial role in the field of refractory materials science and engineering, facilitating the observation and analysis of the material’s pore structure and unique characteristics. However, traditional image restoration techniques have limitations in handling background noise, complex texture restoration, deformation, and dislocation correction, as well as automation and intelligence aspects in restoring material pore structure images. The latest Transformer architecture has shown significant potential in image processing, particularly in image restoration research tasks. Nevertheless, its application in restoring pore structure images of materials remains unexplored. This paper aims to address this gap by focusing on the image restoration of foam ceramic pore structure images using a Transformer-based algorithm for super-resolution (SR) processing. Experimental comparisons with state-of-the-art methods demonstrate a remarkable improvement in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), underscoring the effectiveness of the proposed approach.

Construction and Evaluation of an AI-based CBCT Resolution Optimization Technique for Extracted Teeth

IntroductionIn dental clinical practice, cone-beam computed tomography (CBCT) is commonly used to assist practitioners to recognize the complex morphology of root canal systems; however, because of its resolution limitations, certain small anatomical structures still cannot be accurately recognized on CBCT. The purpose of this study was to perform image super-resolution (SR) processing on CBCT images of extracted human teeth with the help of a deep learning model, and to compare the differences among CBCT, super-resolution computed tomography (SRCT), and micro-computed tomography (Micro-CT) images through three-dimensional reconstruction. MethodsThe deep learning model (Basicvsr++) was selected and modified. The dataset consisted of 171 extracted teeth that met inclusion criteria, with 40 maxillary first molars as the training set and 40 maxillary first molars as well as 91 teeth from other tooth positions as the external test set. The corresponding CBCT, SRCT, and Micro-CT images of each tooth in test sets were reconstructed using Mimics Research 17.0, and the root canal recognition rates in the 3 groups were recorded. The following parameters were measured: volume of hard tissue (V1), volume of pulp chamber and root canal system (V2), length of visible root canals under orifice (VL-X, where X represents the specific root canal), and intersection angle between coronal axis of canal and long axis of tooth (∠X, where X represents the specific root canal). Data were statistically analyzed between CBCT and SRCT images using paired sample t-test and Wilcoxon test analysis, with the measurement from Micro-CT images as the gold standard. ResultsImages from all tested teeth were successfully processed with the SR program. In 4-canal maxillary first molar, identification of MB2 was 72% (18/25) in CBCT group, 92% (23/25) in SRCT group, and 100% (25/25) in Micro-CT group. The difference of hard tissue volume between SRCT and Micro-CT was significantly smaller than that between CBCT and Micro-CT in all tested teeth except 4-canal mandibular first molar (P < .05). Similar results were obtained in volume of pulp chamber and root canal system in all tested teeth (P < .05). As for length of visible root canals under orifice, the difference between SRCT and Micro-CT was significantly smaller than that between CBCT and Micro-CT (P < .05) in most root canals. ConclusionsThe deep learning model developed in this study helps to optimize the root canal morphology of extracted teeth in CBCT. And it may be helpful for the identification of MB2 in the maxillary first molar.

FSN-YOLO: Nearshore Vessel Detection via Fusing Receptive-Field Attention and Lightweight Network

Vessel detection is critical for ensuring maritime transportation and navigational safety, creating a pressing need for detection methodologies that are more efficient, precise, and intelligent in the maritime domain. Nonetheless, accurately detecting vessels across multiple scales remains challenging due to the diversity in vessel types and locations, similarities in ship hull shapes, and disturbances from complex environmental conditions. To address these issues, we introduce an innovative FSN-YOLO framework that utilizes enhanced YOLOv8 with multi-layer attention feature fusion. Specifically, FSN-YOLO employs the backbone structure of FasterNet, enriching feature representations through super-resolution processing with a lightweight Convolutional Neural Network (CNN), thereby achieving a balance between processing speed and model size without compromising accuracy. Furthermore, FSN-YOLO uses the Receptive-Field Attention (RFA) mechanism to adaptively fine-tune the feature responses between channels, significantly boosting the network’s capacity to capture critical information and, in turn, improve the model’s overall performance and enrich the discriminative feature representation of ships. Experimental validation on the Seaship7000 dataset showed that, compared to the baseline YOLOv8l approach, FSN-YOLO considerably increased accuracy, recall rates, and mAP@0.5:0.95 by absolute margins of 0.82%, 1.54%, and 1.56%, respectively, positioning it at the forefront of current state-of-the-art models.

SuperVidConform: Conformation detail-preserving network (CDPN) for video super-resolution

Video Super Resolution (VSR) applications extensively utilize deep learning-based methods. Several VSR methods primarily focus on improving the fine-patterns within reconstructed video frames. It frequently overlooks the crucial aspect of keeping conformation details, particularly sharpness. Therefore, reconstructed video frames often fail to meet expectations. In this paper, we propose a Conformation Detail-Preserving Network (CDPN) named as SuperVidConform. It focuses on restoring local region features and maintaining the sharper details of video frames. The primary focus of this work is to generate the high-resolution (HR) frame from its corresponding low-resolution (LR). It consists of two parts: (i) The proposed model decomposes confirmation details from the ground-truth HR frames to provide additional information for the super-resolution process, and (ii) These video frames pass to the temporal modelling SR network to learn local region features by residual learning that connects the network intra-frame redundancies within video sequences. The proposed approach is designed and validated using VID4, SPMC, and UDM10 datasets. The experimental results show the proposed model presents an improvement of 0.43 dB (VID4), 0.78 dB (SPMC), and 0.84 dB (UDM10) in terms of PSNR. Further, the CDPN model set new standards for the performance of self-generated surveillance datasets.

Automated phase reconstruction and super-resolution with deep learning in digital holography

Digital holography can provide quantitative phase images that are related to the shape and content of biological samples. In particular, high-resolution phase images contain more important details and information about the sample. However, to obtain a high-resolution phase image, various phase reconstruction processes must be performed, while the finite pixel size of the sensor needs to be overcome. We propose a deep learning model that can obtain high-resolution phase images from low-resolution holograms. The proposed model consists of image translation and super-resolution parts, and performs phase reconstruction and the super-resolution process at the same time. We successfully generated sophisticated phase values that closely resembled real images for three scaling factors of (×2, ×3, and ×4). Comparative evaluations with various deep learning models demonstrated the favorable performance of our proposed model. Multi-scale training was also possible, so it was shown that high-resolution phase images could be generated regardless of the scale factor. The proposed model can automatically generate accurate high-resolution phase images from low-resolution holograms, reducing the cost of digital holography, and providing great benefits to biological sample measurement processes.

Manhole Cover Classification Based on Super-Resolution Reconstruction of Unmanned Aerial Vehicle Aerial Imagery

Urban underground pipeline networks are a key component of urban infrastructure, and a large number of older urban areas lack information about their underground pipelines. In addition, survey methods for underground pipelines are often time-consuming and labor-intensive. While the manhole cover serves as the hub connecting the underground pipe network with the ground, the generation of underground pipe network can be realized by obtaining the location and category information of the manhole cover. Therefore, this paper proposed a manhole cover detection method based on UAV aerial photography to obtain ground images, using image super-resolution reconstruction and image positioning and classification. Firstly, the urban image was obtained by UAV aerial photography, and then the YOLOv8 object detection technology was used to accurately locate the manhole cover. Next, the SRGAN network was used to perform super-resolution processing on the manhole cover text to improve the clarity of the recognition image. Finally, the clear manhole cover text image was input into the VGG16_BN network to realize the manhole cover classification. The experimental results showed that the manhole cover classification accuracy of this paper’s method reached 97.62%, which verified its effectiveness in manhole cover detection. The method significantly reduces the time and labor cost and provides a new method for manhole cover information acquisition.

S2CycleDiff: Spatial-Spectral-Bilateral Cycle-Diffusion Framework for Hyperspectral Image Super-resolution

Hyperspectral image super-resolution (HISR) is a technique that can break through the limitation of imaging mechanism to obtain the hyperspectral image (HSI) with high spatial resolution. Although some progress has been achieved by existing methods, most of them directly learn the spatial-spectral joint mapping between the observed images and the target high-resolution HSI (HrHSI), failing to fully reserve the spectral distribution of low-resolution HSI (LrHSI) and the spatial distribution of high-resolution multispectral imagery (HrMSI). To this end, we propose a spatial-spectral-bilateral cycle-diffusion framework (S2CycleDiff) for HISR, which can step-wise generate the HrHSI with high spatial-spectral fidelity by learning the conditional distribution of spatial and spectral super-resolution processes bilaterally. Specifically, a customized conditional cycle-diffusion framework is designed as the backbone to achieve the spatial-spectral-bilateral super-resolution by repeated refinement, wherein the spatial/spectral guided pyramid denoising (SGPD) module seperately takes HrMSI and LrHSI as the guiding factors to achieve the spatial details injection and spectral correction. The outputs of the conditional cycle-diffusion framework are fed into a complementary fusion block to integrate the spatial and spectral details to generate the desired HrHSI. Experiments have been conducted on three widely used datasets to demonstrate the superiority of the proposed method over state-of-the-art HISR methods. The code is available at https://github.com/Jiahuiqu/S2CycleDiff.

Deep Learning Accelerated Brain Diffusion-Weighted MRI with Super Resolution Processing

ObjectivesTo investigate the clinical feasibility and image quality of accelerated brain diffusion-weighted imaging (DL-DWI) with deep learning image reconstruction and super resolution. MethodsEighty-five consecutive patients with clinically indicated MRI at a 3T scanner were prospectively included. Conventional diffusion-weighted data (c-DWI) with four averages were obtained. Reconstructions of one and two averages, as well as DL-DWI, were accomplished. Three experienced readers evaluated the acquired data using a 5-point Likert scale regarding overall image quality, overall contrast, diagnostic confidence, occurrence of artefacts and evaluation of the central region, basal ganglia, brainstem, and cerebellum. To assess interrater agreement, Fleiss’ kappa (ϰ) was determined. Signal intensity (SI) levels for basal ganglia and the central region were estimated via automated segmentation, and SI values of detected pathologies were measured. ResultsIntracranial pathologies were identified in thirty-five patients. DL-DWI was significantly superior for all defined parameters, independently from applied averages (p-value <0.001). Optimum image quality was achieved with DL-DWI by utilizing a single average (p-value <0.001), demonstrating very good (80.9%) to excellent image quality (14.5%) in nearly all cases, compared to 12.5% with very good and 0% with excellent image quality for c-MRI (p-value <0.001). Comparable results could be shown for diagnostic confidence. Inter-rater Fleiss' Kappa demonstrated moderate to substantial agreement for virtually all defined parameters, with good accordance, particularly for the assessment of pathologies (p=0.74). Regarding SI values, no significant difference was found. ConclusionUltra-fast diffusion-weighted imaging with super resolution is feasible, resulting in highly accelerated brain imaging while increasing diagnostic image quality.

Using imaging modalities to predict nanoparticle distribution and treatment efficacy in solid tumors: The growing role of ultrasound.

Nanomedicine in oncology has not had the success in clinical impact that was anticipated in the early stages of the field's development. Ideally, nanomedicines selectively accumulate in tumor tissue and reduce systemic side effects compared to traditional chemotherapeutics. However, this has been more successful in preclinical animal models than in humans. The causes of this failure to translate may be related to the intra- and inter-patient heterogeneity of the tumor microenvironment. Predicting whether a patient will respond positively to treatment prior to its initiation, through evaluation of characteristics like nanoparticle extravasation and retention potential in the tumor, may be a way to improve nanomedicine success rate. While there are many potential strategies to accomplish this, prediction and patient stratification via noninvasive medical imaging may be the most efficient and specific strategy. There have been some preclinical and clinical advances in this area using MRI, CT, PET, and other modalities. An alternative approach that has not been studied as extensively is biomedical ultrasound, including techniques such as multiparametric contrast-enhanced ultrasound (mpCEUS), doppler, elastography, and super-resolution processing. Ultrasound is safe, inexpensive, noninvasive, and capable of imaging the entire tumor with high temporal and spatial resolution. In this work, we summarize the in vivo imaging tools that have been used to predict nanoparticle distribution and treatment efficacy in oncology. We emphasize ultrasound imaging and the recent developments in the field concerning CEUS. The successful implementation of an imaging strategy for prediction of nanoparticle accumulation in tumors could lead to increased clinical translation of nanomedicines, and subsequently, improved patient outcomes. This article is categorized under: Diagnostic Tools In Vivo Nanodiagnostics and Imaging Therapeutic Approaches and Drug Discovery Nanomedicine for Oncologic Disease Therapeutic Approaches and Drug Discovery Emerging Technologies.

Tensor-Based Sparse Representation for Hyperspectral Image Reconstruction Using RGB Inputs

Hyperspectral image (HSI) reconstruction from RGB input has drawn much attention recently and plays a crucial role in further vision tasks. However, current sparse coding algorithms often take each single pixel as the basic processing unit during the reconstruction process, which ignores the strong similarity and relation between adjacent pixels within an image or scene, leading to an inadequate learning of spectral and spatial features in the target hyperspectral domain. In this paper, a novel tensor-based sparse coding method is proposed to integrate both spectral and spatial information represented in tensor forms, which is capable of taking all the neighboring pixels into account during the spectral super-resolution (SSR) process without breaking the semantic structures, thus improving the accuracy of the final results. Specifically, the proposed method recovers the unknown HSI signals using sparse coding on the learned dictionary pairs. Firstly, the spatial information of pixels is used to constrain the sparse reconstruction process, which effectively improves the spectral reconstruction accuracy of pixels. In addition, the traditional two-dimensional dictionary learning is further extended to the tensor domain, by which the structure of inputs can be processed in a more flexible way, thus enhancing the spatial contextual relations. To this end, a rudimentary HSI estimation acquired in the sparse reconstruction stage is further enhanced by introducing the regression method, aiming to eliminate the spectral distortion to some extent. Abundant experiments are conducted on two public datasets, indicating the considerable availability of the proposed framework.

UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration.

Complicated image registration is a key issue in medical image analysis, and deep learning-based methods have achieved better results than traditional methods. The methods include ConvNet-based and Transformer-based methods. Although ConvNets can effectively utilize local information to reduce redundancy via small neighborhood convolution, the limited receptive field results in the inability to capture global dependencies. Transformers can establish long-distance dependencies via a self-attention mechanism; however, the intense calculation of the relationships among all tokens leads to high redundancy. We propose a novel unsupervised image registration method named the unified Transformer and superresolution (UTSRMorph) network, which can enhance feature representation learning in the encoder and generate detailed displacement fields in the decoder to overcome these problems. We first propose a fusion attention block to integrate the advantages of ConvNets and Transformers, which inserts a ConvNet-based channel attention module into a multihead self-attention module. The overlapping attention block, a novel cross-attention method, uses overlapping windows to obtain abundant correlations with match information of a pair of images. Then, the blocks are flexibly stacked into a new powerful encoder. The decoder generation process of a high-resolution deformation displacement field from low-resolution features is considered as a superresolution process. Specifically, the superresolution module was employed to replace interpolation upsampling, which can overcome feature degradation. UTSRMorph was compared to state-of-the-art registration methods in the 3D brain MR (OASIS, IXI) and MR-CT datasets (abdomen, craniomaxillofacial). The qualitative and quantitative results indicate that UTSRMorph achieves relatively better performance. The code and datasets used are publicly available at https://github.com/Runshi-Zhang/UTSRMorph.