Natural Images Research Articles

Neural blind deconvolution for deblurring and supersampling PSMA PET.

Objective. To simultaneously deblur and supersample prostate specific membrane antigen (PSMA) positron emission tomography (PET) images using neural blind deconvolution.Approach. Blind deconvolution is a method of estimating the hypothetical 'deblurred' image along with the blur kernel (related to the point spread function) simultaneously. Traditionalmaximum a posterioriblind deconvolution methods require stringent assumptions and suffer from convergence to a trivial solution. A method of modelling the deblurred image and kernel with independent neural networks, called 'neural blind deconvolution' had demonstrated success for deblurring 2D natural images in 2020. In this work, we adapt neural blind deconvolution to deblur PSMA PET images while simultaneous supersampling to double the original resolution. We compare this methodology with several interpolation methods in terms of resultant blind image quality metrics and test the model's ability to predict accurate kernels by re-running the model after applying artificial 'pseudokernels' to deblurred images. The methodology was tested on a retrospective set of 30 prostate patients as well as phantom images containing spherical lesions of various volumes.Main results. Neural blind deconvolution led to improvements in image quality over other interpolation methods in terms of blind image quality metrics, recovery coefficients, and visual assessment. Predicted kernels were similar between patients, and the model accurately predicted several artificially-applied pseudokernels. Localization of activity in phantom spheres was improved after deblurring, allowing small lesions to be more accurately defined.Significance. The intrinsically low spatial resolution of PSMA PET leads to partial volume effects (PVEs) which negatively impact uptake quantification in small regions. The proposed method can be used to mitigate this issue, and can be straightforwardly adapted for other imaging modalities.

Enhancing fracture diagnosis in pelvic X-rays by deep convolutional neural network with synthesized images from 3D-CT

Pelvic fractures pose significant challenges in medical diagnosis due to the complex structure of the pelvic bones. Timely diagnosis of pelvic fractures is critical to reduce complications and mortality rates. While computed tomography (CT) is highly accurate in detecting pelvic fractures, the initial diagnostic procedure usually involves pelvic X-rays (PXR). In recent years, many deep learning-based methods have been developed utilizing ImageNet-based transfer learning for diagnosing hip and pelvic fractures. However, the ImageNet dataset contains natural RGB images which are different than PXR. In this study, we proposed a two-step transfer learning approach that improved the diagnosis of pelvic fractures in PXR images. The first step involved training a deep convolutional neural network (DCNN) using synthesized PXR images derived from 3D-CT by digitally reconstructed radiographs (DRR). In the second step, the classification layers of the DCNN were fine-tuned using acquired PXR images. The performance of the proposed method was compared with the conventional ImageNet-based transfer learning method. Experimental results demonstrated that the proposed DRR-based method, using 20 synthesized PXR images for each CT, achieved superior performance with the area under the receiver operating characteristic curves (AUROCs) of 0.9327 and 0.8014 for visible and invisible fractures, respectively. The ImageNet-based method yields AUROCs of 0.8908 and 0.7308 for visible and invisible fractures, respectively.

Automatic Marker Generation to Similarity Based Region Merging Algorithm using Edge Information

One of the most important tasks in computer vision is image segmentation. Several interactive strategies are utilized for picture segmentation because automated techniques are difficult for this kind of work. The outcome of interactive methods is mostly determined by user input. Obtaining excellent interactions for huge datasets is challenging. However, automated picture segmentation is starting to play a significant role in image analysis and computer vision. Effective and efficient segmentation outcomes are obtained using the interacting region merging method suggested by Maximal Similarity based region merging algorithm. Limitation of MSBRM is it requires some efforts on the part of users and is yet not a fully-automatic approach. Here, we suggest a completely fresh unsupervised image segmentation method that combines edge information with MSBRM. We propose an integrated framework to generate object markers for similarity based region algorithm using edge information. Long edges give rough distribution of objects in image. After retrieving edges using phase congruency, edge processing operations are employed to remove small edges and to group color similar long boundaries. Centroids of long boundaries are used as object markers to the MSBRM algorithm. The generation of object markers is done using edge segment grouping. These object markers guide the region merging process. The proposed method shows its effectiveness in segmenting natural real world color images.

A medical image segmentation method for rectal tumors based on multi-scale feature retention and multiple attention mechanisms.

With the continuous development of deep learning algorithms in the field of medical images, models for medical image processing based on convolutional neural networks have made great progress. Since medical images of rectal tumors are characterized by specific morphological features and complex edges that differ from natural images, achieving good segmentation results often requires a higher level of enrichment through the utilization of semantic features. The efficiency of feature extraction and utilization has been improved to some extent through enhanced hardware arithmetic and deeper networks in most models. However, problems still exist with detail loss and difficulty in feature extraction, arising from the extraction of high-level semantic features in deep networks. In this work, a novel medical image segmentation model has been proposed for Magnetic Resonance Imaging (MRI) image segmentation of rectal tumors. The model constructs a backbone architecture based on the idea of jump-connected feature fusion and solves the problems of detail feature loss and low segmentation accuracy using three novel modules: Multi-scale Feature Retention (MFR), Multi-branch Cross-channel Attention (MCA), and Coordinate Attention (CA). Compared with existing methods, our proposed model is able to segment the tumor region more effectively, achieving 97.4% and 94.9% in Dice and mIoU metrics, respectively, exhibiting excellent segmentation performance and computational speed. Our proposed model has improved the accuracy of both lesion region and tumor edge segmentation. In particular, the determination of the lesion region can help doctors identify the tumor location in clinical diagnosis, and the accurate segmentation of the tumor edge can assist doctors in judging the necessity and feasibility of surgery.

Utilizing Diagnostic Musculoskeletal Ultrasound in the Evaluation of Cubital Tunnel Syndrome and Ulnar Nerve Pathology.

Cubital Tunnel Syndrome (CTS), is a frequently encountered condition in clinical settings, typically arising from static or dynamic compression of the ulnar nerve. CTS is characterized by the irritation of the ulnar nerve within the cubital tunnel and is a common neuropathic condition that can significantly impact functional abilities and quality of life. This article explores the utility of diagnostic musculoskeletal ultrasound (MSK-US) in the evaluation of the cubital tunnel and the ulnar nerve. The cubital tunnel syndrome, a common entrapment neuropathy of the ulnar nerve, often presents diagnostic challenges. Traditionally, nerve conduction studies and electromyography have been the primary diagnostic tools used to diagnose CTS. Given the superficial nature of the ulnar groove and its adjacent structures, MSK-US imaging known for its non-invasive nature and high-resolution imaging capabilities, has become an increasingly valuable tool in the assessment and management of various musculoskeletal conditions. This article reviews the technical aspects of MSK-US, its advantages in visualizing the ulnar nerve and related structures, and its implications in clinical practice for rehabilitation providers. By integrating MSK-US in the evaluation of CTS, rehabilitation professionals can enhance diagnostic accuracy, tailor interventions, and improve patient outcomes.

DeepTree: Pathological Image Classification Through Imitating Tree-Like Strategies of Pathologists.

Digitization of pathological slides has promoted the research of computer-aided diagnosis, in which artificial intelligence analysis of pathological images deserves attention. Appropriate deep learning techniques in natural images have been extended to computational pathology. Still, they seldom take into account prior knowledge in pathology, especially the analysis process of lesion morphology by pathologists. Inspired by the diagnosis decision of pathologists, we design a novel deep learning architecture based on tree-like strategies called DeepTree. It imitates pathological diagnosis methods, designed as a binary tree structure, to conditionally learn the correlation between tissue morphology, and optimizes branches to finetune the performance further. To validate and benchmark DeepTree, we build a dataset of frozen lung cancer tissues and design experiments on a public dataset of breast tumor subtypes and our dataset. Results show that the deep learning architecture based on tree-like strategies makes the pathological image classification more accurate, transparent, and convincing. Simultaneously, prior knowledge based on diagnostic strategies yields superior representation ability compared to alternative methods. Our proposed methodology helps improve the trust of pathologists in artificial intelligence analysis and promotes the practical clinical application of pathology-assisted diagnosis.

Impression Generation From X-ray Images Using Machine Learning

Due to the incorporation of machine learning techniques, medical image analysis has made significant strides in recent years. In this paper, we concentrate on a crucial application: employing machine learning to extract literary impressions from chest X-ray pictures. The goal of the project is to fill the gap between natural language processing and medical imaging by enabling the automatic creation of radio-logical impressions, which are crucial for diagnostic reports. The problem statement includes a number of significant difficulties. A lack of positive examples compared to negative cases in medical datasets can skew model training and impair diagnostic precision. For impression formation, accurate feature extraction from chest X-ray pictures is essential. The CheXNet model must be carefully adjusted to the particular task at hand in order to be used for this purpose. Converting visual information into cohesive literary perceptions is the main challenge. To do this while minimizing the risk of overfitting, a sequence-to-sequence model with an attention mechanism must be used to precisely match image attributes with textual information. These difficulties highlight the project’s complexity and highlight the requirement for exact machine learning methods, advanced architectural layouts, and strategic feature extraction in order to enable the automated generation of high-quality radio-logical images.

SurgNet: Self-Supervised Pretraining With Semantic Consistency for Vessel and Instrument Segmentation in Surgical Images.

Blood vessel and surgical instrument segmentation is a fundamental technique for robot-assisted surgical navigation. Despite the significant progress in natural image segmentation, surgical image-based vessel and instrument segmentation are rarely studied. In this work, we propose a novel self-supervised pretraining method (SurgNet) that can effectively learn representative vessel and instrument features from unlabeled surgical images. As a result, it allows for precise and efficient segmentation of vessels and instruments with only a small amount of labeled data. Specifically, we first construct a region adjacency graph (RAG) based on local semantic consistency in unlabeled surgical images and use it as a self-supervision signal for pseudo-mask segmentation. We then use the pseudo-mask to perform guided masked image modeling (GMIM) to learn representations that integrate structural information of intraoperative objectives more effectively. Our pretrained model, paired with various segmentation methods, can be applied to perform vessel and instrument segmentation accurately using limited labeled data for fine-tuning. We build an Intraoperative Vessel and Instrument Segmentation (IVIS) dataset, comprised of ~3 million unlabeled images and over 4,000 labeled images with manual vessel and instrument annotations to evaluate the effectiveness of our self-supervised pretraining method. We also evaluated the generalizability of our method to similar tasks using two public datasets. The results demonstrate that our approach outperforms the current state-of-the-art (SOTA) self-supervised representation learning methods in various surgical image segmentation tasks.

From Composited to Real-World: Transformer-Based Natural Image Matting

From Composited to Real-World: Transformer-Based Natural Image Matting

COVID-19 Image Classification: A Comparative Performance Analysis of Hand-Crafted vs. Deep Features

This study investigates techniques for medical image classification, specifically focusing on COVID-19 scans obtained through computer tomography (CT). Firstly, handcrafted methods based on feature engineering are explored due to their suitability for training traditional machine learning (TML) classifiers (e.g., Support Vector Machine (SVM)) when faced with limited medical image datasets. In this context, I comprehensively evaluate and compare 27 descriptor sets. More recently, deep learning (DL) models have successfully analyzed and classified natural and medical images. However, the scarcity of well-annotated medical images, particularly those related to COVID-19, presents challenges for training DL models from scratch. Consequently, I leverage deep features extracted from 12 pre-trained DL models for classification tasks. This work presents a comprehensive comparative analysis between TML and DL approaches in COVID-19 image classification.

Gan-based data augmentation to improve breast ultrasound and mammography mass classification

Data imbalance is a common problem in breast cancer diagnosis, to address this challenge, the research explores the use of Generative Adversarial Networks (GANs) to generate synthetic medical data. Various GAN methods, including Wasserstein GAN with Gradient Penalty (WGAN-GP), Cycle GAN, Conditional GAN, and Spectral Normalization GAN (SNGAN), were tested for data augmentation in breast regions of interest (ROIs) using mammography and ultrasound databases. The study employed real, synthetic, and hybrid ROIs (128x128 pixels) to train a Resnet network for classifying as benign (B) or malignant (M) classes. The quality and diversity of the synthetic data were assessed using several metrics: Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Structural Similarity Index (SSIM), Multi-Scale SSIM (MS-SSIM), Blind Reference Image Spatial Quality Evaluator (BRISQUE), Naturalness Image Quality Evaluator (NIQE), and Perception-based Image Quality Evaluator (PIQE).Results revealed that the SNGAN model (FID = 52.89) was most effective for augmenting mammography data, while CGAN (FID = 116.03) excelled with ultrasound data. Cycle GAN and WGAN-GP, though demonstrating lower KID values, did not perform better than SNGAN and CGAN. The lower average MS-SSIM values suggested that SNGAN and CGAN produced a high diversity of synthetic images. However, lower SSIM, BRISQUE, NIQE, and PIQE values indicated poor quality in both real and synthetic images. Classification results showed high accuracy without data augmentation in both US (93.1 %B/94.9 %M) and mammography (80.9 %B/76.9 %M). The research concludes that preprocessing and characterizing ROIs by abnormality type is crucial to generate diverse synthetic data and improve accuracy in the classification process using combined GANs and CNN models.

Supervised representation learning based on various levels of pediatric radiographic views for transfer learning

Transfer learning plays a pivotal role in addressing the paucity of data, expediting training processes, and enhancing model performance. Nonetheless, the prevailing practice of transfer learning predominantly relies on pre-trained models designed for the natural image domain, which may not be well-suited for the medical image domain in grayscale. Recognizing the significance of leveraging transfer learning in medical research, we undertook the construction of class-balanced pediatric radiograph datasets collectively referred to as PedXnets, grounded in radiographic views using the pediatric radiographs collected over 24 years at Asan Medical Center. For PedXnets pre-training, approximately 70,000 X-ray images were utilized. Three different pre-training weights of PedXnet were constructed using Inception V3 for various radiation perspective classifications: Model-PedXnet-7C, Model-PedXnet-30C, and Model-PedXnet-68C. We validated the transferability and positive effects of transfer learning of PedXnets through pediatric downstream tasks including fracture classification and bone age assessment (BAA). The evaluation of transfer learning effects through classification and regression metrics showed superior performance of Model-PedXnets in quantitative assessments. Additionally, visual analyses confirmed that the Model-PedXnets were more focused on meaningful regions of interest.

Automatic landslide detection and visualization by using deep ensemble learning method

Rapid detection of damages occurring as a result of natural disasters is vital for emergency response. In recent years, remote sensing techniques have been commonly used for the automatic categorization and localization of such events using satellite images. Trained based on natural disaster images, a convolutional neural network (CNN) has been applied as a highly successful method, with its ability to reveal outstanding features. Studies aiming to detect target points obtained as a result of extracting visual features from natural images within these networks have achieved their goals. In this study, ensemble learning methods have been suggested as a means to develop the detection of landslide areas from landslide satellite images. Landslide image dataset has been trained for their categorization in CNN models and then they have been used again to localize landslide regions. While model predictions develop overall performance and status, different ensemble strategies have been used and integrated to reduce the sensitivity to prediction variance and training data. Class-selective relevance mapping (CRM) has been used to visualize individual CNN models and ensemble learned behaviors. As a result of the comparisons made based on mean average precision metrics and the criteria of intersection over union, model ensembles have proved to show higher localization performance than any other individual model.

Zero-Shot Sketch-Based Image Retrieval Using StyleGen and Stacked Siamese Neural Networks.

Sketch-based image retrieval (SBIR) refers to a sub-class of content-based image retrieval problems where the input queries are ambiguous sketches and the retrieval repository is a database of natural images. In the zero-shot setup of SBIR, the query sketches are drawn from classes that do not match any of those that were used in model building. The SBIR task is extremely challenging as it is a cross-domain retrieval problem, unlike content-based image retrieval problems because sketches and images have a huge domain gap. In this work, we propose an elegant retrieval methodology, StyleGen, for generating fake candidate images that match the domain of the repository images, thus reducing the domain gap for retrieval tasks. The retrieval methodology makes use of a two-stage neural network architecture known as the stacked Siamese network, which is known to provide outstanding retrieval performance without losing the generalizability of the approach. Experimental studies on the image sketch datasets TU-Berlin Extended and Sketchy Extended, evaluated using the mean average precision (mAP) metric, demonstrate a marked performance improvement compared to the current state-of-the-art approaches in the domain.

Tensorial Evolutionary Optimization for Natural Image Matting

Natural image matting has garnered increasing attention in various computer vision applications. The matting problem aims to find the optimal foreground/background (F/B) color pair for each unknown pixel and thus obtain an alpha matte indicating the opacity of the foreground object. This problem is typically modeled as a large-scale pixel pair combinatorial optimization (PPCO) problem. Heuristic optimization is widely employed to tackle the PPCO problem owing to its gradient-free property and promising search ability. However, traditional heuristic methods often encode F/B solutions to a one-dimensional (1D) representation and then evolve the solutions in a 1D manner. This 1D representation destroys the intrinsic two-dimensional (2D) structure of images, where the significant spatial correlations among pixels are ignored. Moreover, the 1D representation also brings operation inefficiency. To address the above issues, this article develops a spatial-aware tensorial evolutionary image matting (TEIM) method. Specifically, the matting problem is modeled as a 2D Spatial-PPCO (S-PPCO) problem, and a global tensorial evolutionary optimizer is proposed to tackle the S-PPCO problem. The entire population is represented as a whole by a third-order tensor, in which individuals are classified into two types: F and B individuals for denoting the 2D F/B solutions, respectively. The evolution process, consisting of three tensorial evolutionary operators, is implemented based on pure tensor computation for efficiently seeking F/B solutions. The local spatial smoothness of images is also integrated into the evaluation process for obtaining a high-quality alpha matte. Experimental results compared with state-of-the-art methods validate the effectiveness of TEIM.

Calibrating Bayesian Decoders of Neural Spiking Activity.

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Zero-shot sketch-based image retrieval via adaptive relation-aware metric learning

Retrieving natural images with the query sketches under the zero-shot scenario is known as zero-shot sketch-based image retrieval (ZS-SBIR). Most of the best-performing methods adapt the triplet loss to learn projections that map natural images and sketches to a latent embedding space. They nevertheless neglect the modality gap between the hand-drawn sketches and the photos and consider no difference between any two incorrect classes, which limits their performance in real use cases. Towards this end, we put forward a simple and effective model, which adopts relation-aware metric learning to suppress the modality gap between the sketches and the photos. We also propose an adaptive margin that utilizes each anchor in embedding space to improve clustering ability in metric learning. Extensive experiments on the Sketchy and TU-Berlin datasets show the dominant position of our proposed model over SOTA competitors.

Physically Motivated Deep Learning to Superresolve and Cross Calibrate Solar Magnetograms

Superresolution (SR) aims to increase the resolution of images by recovering detail. Compared to standard interpolation, deep learning-based approaches learn features and their relationships to leverage prior knowledge of what low-resolution patterns look like in higher resolution. Deep neural networks can also perform image cross-calibration by learning the systematic properties of the target images. While SR for natural images aims to create perceptually convincing results, SR of scientific data requires careful quantitative evaluation. In this work, we demonstrate that deep learning can increase the resolution and calibrate solar imagers belonging to different instrumental generations. We convert solar magnetic field images taken by the Michelson Doppler Imager (resolution ∼2″ pixel−1; space based) and the Global Oscillation Network Group (resolution ∼2.″5 pixel−1; ground based) to the characteristics of the Helioseismic and Magnetic Imager (resolution ∼0.″5 pixel−1; space based). We also establish a set of performance measurements to benchmark deep-learning-based SR and calibration for scientific applications.

Chinese Ancient Paintings Inpainting Based on Edge Guidance and Multi-Scale Residual Blocks

Chinese paintings have great cultural and artistic significance and are known for their delicate lines and rich textures. Unfortunately, many ancient paintings have been damaged due to historical and natural factors. The deep learning methods that are successful in restoring natural images cannot be applied to the inpainting of ancient paintings. Thus, we propose a model named Edge-MSGAN for inpainting Chinese ancient paintings based on edge guidance and multi-scale residual blocks. The Edge-MSGAN utilizes edge images to direct the completion network in order to generate entire ancient paintings. It then applies the multi-branch color correction network to adjust the colors. Furthermore, the model uses multi-scale channel attention residual blocks to learn the semantic features of ancient paintings at various levels. At the same time, by using polarized self-attention, the model can improve its concentration on significant structures, edges, and details, which leads to paintings that possess clear lines and intricate details. Finally, we have created a dataset for ancient paintings inpainting, and have conducted experiments in order to evaluate the model’s performance. After comparing the proposed model with state-of-the-art models from qualitative and quantitative aspects, it was found that our model is better at inpainting the texture, edge, and color of ancient paintings. Therefore, our model achieved maximum PSNR and SSIM values of 34.7127 and 0.9280 respectively, and minimum MSE and LPIPS values of 0.0006 and 0.0495, respectively.

Back to basics: Fast denoising iterative algorithm

We introduce Back to Basics (BTB), a fast iterative algorithm for noise reduction. Our method is computationally efficient, does not require training or ground truth data, and can be applied in the presence of independent noise, as well as correlated (coherent) noise, where the noise level is unknown. We examine three study cases: natural image denoising in the presence of additive white Gaussian noise, Poisson-distributed image denoising, and speckle suppression in optical coherence tomography (OCT). Experimental results demonstrate that the proposed approach can effectively improve image quality, in challenging noise settings. Theoretical guarantees are provided for convergence stability.