Corrupted Images Research Articles

The Role of Deep Learning in Medical Image Inpainting: A Systematic Review

Image inpainting is a crucial technique in computer vision, particularly for reconstructing corrupted images. In medical imaging, it addresses issues from instrumental errors, artifacts, or human factors. The development of deep learning techniques has revolutionized image inpainting, allowing for the generation of high-level semantic information to ensure structural and textural consistency in restored images. This paper presents a comprehensive review of 53 studies on deep image inpainting in medical imaging, analyzing its evolution, impact, and limitations. The findings highlight the significance of deep image inpainting in artifact removal and enhancing the performance of multi-task approaches by localizing and inpainting regions of interest. Furthermore, the study identifies magnetic resonance imaging and computed tomography as the predominant modalities and highlights generative adversarial networks and U-Net as preferred architectures. Future research directions include the development of blind inpainting techniques, the exploration of techniques suitable for 3D/4D images, multiple artifacts, and multi-task applications, and the improvement of architectures.

Benchmarking the Robustness of Instance Segmentation Models.

This article presents a comprehensive evaluation of instance segmentation models with respect to real-world image corruptions as well as out-of-domain image collections, e.g., images captured by a different set-up than the training dataset. The out-of-domain image evaluation shows the generalization capability of models, an essential aspect of real-world applications, and an extensively studied topic of domain adaptation. These presented robustness and generalization evaluations are important when designing instance segmentation models for real-world applications and picking an off-the-shelf pretrained model to directly use for the task at hand. Specifically, this benchmark study includes state-of-the-art network architectures, network backbones, normalization layers, models trained starting from scratch versus pretrained networks, and the effect of multitask training on robustness and generalization. Through this study, we gain several insights. For example, we find that group normalization (GN) enhances the robustness of networks across corruptions where the image contents stay the same but corruptions are added on top. On the other hand, batch normalization (BN) improves the generalization of the models across different datasets where statistics of image features change. We also find that single-stage detectors do not generalize well to larger image resolutions than their training size. On the other hand, multistage detectors can easily be used on images of different sizes. We hope that our comprehensive study will motivate the development of more robust and reliable instance segmentation models.

Will Transformers change gastrointestinal endoscopic image analysis? A comparative analysis between CNNs and Transformers, in terms of performance, robustness and generalization

Gastrointestinal endoscopic image analysis presents significant challenges, such as considerable variations in quality due to the challenging in-body imaging environment, the often-subtle nature of abnormalities with low interobserver agreement, and the need for real-time processing. These challenges pose strong requirements on the performance, generalization, robustness and complexity of deep learning-based techniques in such safety–critical applications. While Convolutional Neural Networks (CNNs) have been the go-to architecture for endoscopic image analysis, recent successes of the Transformer architecture in computer vision raise the possibility to update this conclusion. To this end, we evaluate and compare clinically relevant performance, generalization and robustness of state-of-the-art CNNs and Transformers for neoplasia detection in Barrett’s esophagus. We have trained and validated several top-performing CNNs and Transformers on a total of 10,208 images (2,079 patients), and tested on a total of 7,118 images (998 patients) across multiple test sets, including a high-quality test set, two internal and two external generalization test sets, and a robustness test set. Furthermore, to expand the scope of the study, we have conducted the performance and robustness comparisons for colonic polyp segmentation (Kvasir-SEG) and angiodysplasia detection (Giana). The results obtained for featured models across a wide range of training set sizes demonstrate that Transformers achieve comparable performance as CNNs on various applications, show comparable or slightly improved generalization capabilities and offer equally strong resilience and robustness against common image corruptions and perturbations. These findings confirm the viability of the Transformer architecture, particularly suited to the dynamic nature of endoscopic video analysis, characterized by fluctuating image quality, appearance and equipment configurations in transition from hospital to hospital. The code is made publicly available at: https://github.com/BONS-AI-VCA-AMC/Endoscopy-CNNs-vs-Transformers.

Image dejittering on the perspective of spatially-varying mixed noise removal

Jittery image is visually abnormal in jags of edge and loss of coherence. The problem of image dejittering is challenging to resolve due to the ubiquitous blur and/or noise in jittery data. In this paper, we devote to the pixel-jitter (possibly blurry) image recovery on the perspective of spatially-varying mixed noise removal. By viewing jittery image as the corruption of ideal image with outliers and spatially-varying Gaussian noise, we proposed a two-phase (including filtering and diffusing phases) image dejittering approach. In the filtering phase, outliers posed by jitters around edges are inspected by median filters. In the diffusing phase, structure tensor based anisotropic diffusion is exploited to reduce the perturbations in piecewise smooth image regions. Upon the spectral decomposition of structure tensor, the variational model in diffusing phase can be solved by some state-of-the-art optimization methods. Numerical simulations on synthetic and real jittery data demonstrate the compelling performance of the proposed approach. The Matlab source codes of the proposed approach are available at the repositories of https://github.com/WenxingZhang.

Benchmarking robustness of deep neural networks in semantic segmentation of fluorescence microscopy images

BackgroundFluorescence microscopy (FM) is an important and widely adopted biological imaging technique. Segmentation is often the first step in quantitative analysis of FM images. Deep neural networks (DNNs) have become the state-of-the-art tools for image segmentation. However, their performance on natural images may collapse under certain image corruptions or adversarial attacks. This poses real risks to their deployment in real-world applications. Although the robustness of DNN models in segmenting natural images has been studied extensively, their robustness in segmenting FM images remains poorly understoodResultsTo address this deficiency, we have developed an assay that benchmarks robustness of DNN segmentation models using datasets of realistic synthetic 2D FM images with precisely controlled corruptions or adversarial attacks. Using this assay, we have benchmarked robustness of ten representative models such as DeepLab and Vision Transformer. We find that models with good robustness on natural images may perform poorly on FM images. We also find new robustness properties of DNN models and new connections between their corruption robustness and adversarial robustness. To further assess the robustness of the selected models, we have also benchmarked them on real microscopy images of different modalities without using simulated degradation. The results are consistent with those obtained on the realistic synthetic images, confirming the fidelity and reliability of our image synthesis method as well as the effectiveness of our assay.ConclusionsBased on comprehensive benchmarking experiments, we have found distinct robustness properties of deep neural networks in semantic segmentation of FM images. Based on the findings, we have made specific recommendations on selection and design of robust models for FM image segmentation.

A Review on FPGA-based Architectures for Noise Removal of Digital Images using High-Level Design

Image corruption during the acquisition and transmission operations may be caused by a variety of disturbances. Scientists are still having trouble getting rid of the noise in the original picture. picture denoising is a technique used to improve a picture's visual quality and extract small details from a damaged image. Improving an image to make it crisper than it was originally is the aim of image restoration. The field of image processing is vast. This paper presents the results of some significant work in the subject of image denoising. It looks into image denoising using some of the most recent methods, such as SEPD (Simple Edge Preserved De-noising) and RSEPD (Reduced Simple Edge Preserved De- noising).Image corruption during the acquisition and transmission operations may be caused by a variety of disturbances. Scientists are still having trouble getting rid of the noise in the original picture. picture denoising is a technique used to improve a picture's visual quality and extract small details from a damaged image. Improving an image to make it crisper than it was originally is the aim of image restoration. The field of image processing is vast. This paper presents the results of some significant work in the subject of image denoising. It looks into image denoising using some of the most recent methods, such as SEPD (Simple Edge Preserved De-noising) and RSEPD (Reduced Simple Edge Preserved De- noising). Keywords— Power consumption analysis, Digital logic circuits, Ultra-low power, Subthreshold operation, Static CMOS gates

Transforming experimental radiology: Design and implementation of an innovative ePACS image storage system for AI imaging research environments

Introduction and purpose: We present the needs, design, development, implementation, and accessibility of a crafted experimental PACS (ePACS) system to securely store images, ensuring efficiency and ease of use for AI processing, specifically tailored for research scenarios, including phantoms, animal and human studies and quality assurance (QA) exams. The ePACS system plays a crucial role in any medical imaging departments that handle non-care profile studies, such as protocol adjustments and dummy runs. By effectively segregating non-care profile studies from the healthcare assistance, the ePACS usefully prevents errors both in clinical practice and storage security. Methods and results: The developed ePACS system considers the best practices for management, maintenance, access, long-term storage and backups, regulatory audits, and economic aspects. Moreover, key aspects of the ePACS system include the design of data flows with a focus on incorporating data security and privacy, access control and levels based on user profiles, internal data management policies, standardized architecture, infrastructure and application monitorization and traceability, and periodic backup policies. A new tool called DicomStudiesQA has been developed to standardize the analysis of DICOM studies. The tool automatically identifies, extracts, and renames series using a consistent nomenclature. It also detects corrupted images and merges separated dynamic series that were initially split, allowing for streamlined post-processing. Discussion and conclusions: The developed ePACS system encompasses a successful implementation, both in hospital and research environments, showcasing its transformative nature and the challenging yet crucial transfer of knowledge to industry. This underscores the practicality and real-world applicability of our innovative approach, highlighting the significant impact it has on the field of experimental radiology.

Benchmarking PathCLIP for Pathology Image Analysis.

Accurate image classification and retrieval are of importance for clinical diagnosis and treatment decision-making. The recent contrastive language-image pre-training (CLIP) model has shown remarkable proficiency in understanding natural images. Drawing inspiration from CLIP, pathology-dedicated CLIP (PathCLIP) has been developed, utilizing over 200,000 image and text pairs in training. While the performance the PathCLIP is impressive, its robustness under a wide range of image corruptions remains unknown. Therefore, we conduct an extensive evaluation to analyze the performance of PathCLIP on various corrupted images from the datasets of osteosarcoma and WSSS4LUAD. In our experiments, we introduce eleven corruption types including brightness, contrast, defocus, resolution, saturation, hue, markup, deformation, incompleteness, rotation, and flipping at various settings. Through experiments, we find that PathCLIP surpasses OpenAI-CLIP and the pathology language-image pre-training (PLIP) model in zero-shot classification. It is relatively robust to image corruptions including contrast, saturation, incompleteness, and orientation factors. Among the eleven corruptions, hue, markup, deformation, defocus, and resolution can cause relatively severe performance fluctuation of the PathCLIP. This indicates that ensuring the quality of images is crucial before conducting a clinical test. Additionally, we assess the robustness of PathCLIP in the task of image-to-image retrieval, revealing that PathCLIP performs less effectively than PLIP on osteosarcoma but performs better on WSSS4LUAD under diverse corruptions. Overall, PathCLIP presents impressive zero-shot classification and retrieval performance for pathology images, but appropriate care needs to be taken when using it.

Semi-hard constraint augmentation of triplet learning to improve image corruption classification

Semi-hard constraint augmentation of triplet learning to improve image corruption classification

Automated MRI Quality Assessment of Brain T1-weighted MRI in Clinical Data Warehouses: A Transfer Learning Approach Relying on Artefact Simulation

The emergence of clinical data warehouses (CDWs), which contain the medical data of millions of patients, has paved the way for vast data sharing for research. The quality of MRIs gathered in CDWs differs greatly from what is observed in research settings and reflects a certain clinical reality. Consequently, a significant proportion of these images turns out to be unusable due to their poor quality. Given the massive volume of MRIs contained in CDWs, the manual rating of image quality is impossible. Thus, it is necessary to develop an automated solution capable of effectively identifying corrupted images in CDWs. This study presents an innovative transfer learning method for automated quality con- trol of 3D gradient echo T1-weighted brain MRIs within a CDW, leveraging artefact sim- ulation. We first intentionally corrupt images from research datasets by inducing poorer contrast, adding noise and introducing motion artefacts. Subsequently, three artefact- specific models are pre-trained using these corrupted images to detect distinct types of artefacts. Finally, the models are generalised to routine clinical data through a transfer learning technique, utilising 3660 manually annotated images. The overall image quality is inferred from the results of the three models, each designed to detect a specific type of artefact. Our method was validated on an independent test set of 385 3D gradient echo T1-weighted MRIs. Our proposed approach achieved excellent results for the detection of bad quality MRIs, with a balanced accuracy of over 87%, surpassing our previous approach by 3.5 percent points. Additionally, we achieved a satisfactory balanced accuracy of 79% for the detection of moderate quality MRIs, outperforming our previous performance by 5 percent points. Our framework provides a valuable tool for exploiting the potential of MRIs in CDWs.

STATNet: One-stage coal-gangue detector based on deep learning algorithm for real industrial application

Coal-gangue object detection has attracted substantial attention because it is the core of realizing vision-based intelligent and green coal separation. However, most existing studies have been focused on laboratory datasets and prioritized model lightweight. This makes the coal-gangue object detection challenging to adapt to the complex and harsh scenes of real production environments. Therefore, our project collected and labeled image datasets of coal and gangue under real production conditions from a coal preparation plant. We then designed a one-stage object model, named STATNet, following the “backbone-neck-head” architecture with the aim of enhancing the detection accuracy under industrial coal preparation scenarios. The proposed model utilizes Swin Transformer as backbone module to extract multi-scale features, improved path augmentation feature pyramid network (iPAFPN) as neck module to enrich feature fusion, and task-aligned head (TAH) as head module to mitigate conflicts and misalignments between classification and localization tasks. Experimental results on a real-world industrial dataset demonstrate that the proposed STATNet model achieves an impressive AP50 of 89.27 %, significantly surpassing several state-of-the-art baseline models by 2.02 % to 5.58 %. Additionally, it exhibits stronger robustness in resisting image corruption and perturbation. These findings demonstrate its promising prospects in practical coal and gangue separation applications.

Real-World Video Super-Resolution with a Degradation-Adaptive Model.

Video super-resolution (VSR) remains challenging for real-world applications due to complex and unknown degradations. Existing methods lack the flexibility to handle video sequences with different degradation levels, thus failing to reflect real-world scenarios. To address this problem, we propose a degradation-adaptive video super-resolution network (DAVSR) based on a bidirectional propagation network. Specifically, we adaptively employ three distinct degradation levels to process input video sequences, aiming to obtain training pairs that reflect a variety of real-world corrupted images. We also equip the network with a pre-cleaning module to reduce noise and artifacts in the low-quality video sequences prior to information propagation. Additionally, compared to previous flow-based methods, we employ an unsupervised optical flow estimator to acquire a more precise optical flow to guide inter-frame alignment. Meanwhile, while maintaining network performance, we streamline the propagation network branches and the structure of the reconstruction module of the baseline network. Experiments are conducted on datasets with diverse degradation types to validate the effectiveness of DAVSR. Our method exhibits an average improvement of 0.18 dB over a recent SOTA approach (DBVSR) in terms of the PSNR metric. Extensive experiments demonstrate the effectiveness of our network in handling real-world video sequences with different degradation levels.

Toward Blind Flare Removal Using Knowledge-Driven Flare-Level Estimator.

Lens flare is a common phenomenon when strong light rays arrive at the camera sensor and a clean scene is consequently mixed up with various opaque and semi-transparent artifacts. Existing deep learning methods are always constrained with limited real image pairs for training. Though recent synthesis-based approaches are found effective, synthesized pairs still deviate from the real ones as the mixing mechanism of flare artifacts and scenes in the wild always depends on a line of undetermined factors, such as lens structure, scratches, etc. In this paper, we present a new perspective from the blind nature of the flare removal task in a knowledge-driven manner. Specifically, we present a simple yet effective flare-level estimator to predict the corruption level of a flare-corrupted image. The estimated flare-level can be interpreted as additive information of the gap between corrupted images and their flare-free correspondences to facilitate a network at both training and testing stages adaptively. Besides, we utilize a flare-level modulator to better integrate the estimations into networks. We also devise a flare-aware block for more accurate flare recognition and reconstruction. Additionally, we collect a new real-world flare dataset for benchmarking, namely WiderFlare. Extensive experiments on three benchmark datasets demonstrate that our method outperforms state-of-the-art methods quantitatively and qualitatively.

Unified Multi-Modal Image Synthesis for Missing Modality Imputation.

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

Survey: Image mixing and deleting for data augmentation

Neural networks are prone to overfitting and memorizing data patterns. To avoid over-fitting and enhance their generalization and performance, various methods have been suggested in the literature, including dropout, regularization, label smoothing, etc. One such method is augmentation which introduces different types of corruption in the data to prevent the model from overfitting and to memorize patterns present in the data. A sub-area of data augmentation is image mixing and deleting. This specific type of augmentation either deletes image regions or mixes two images to hide or make particular characteristics of images confusing for the network, forcing it to emphasize the overall structure of the object in an image. Models trained with this approach have proven to perform and generalize well compared to those trained without image mixing or deleting. An added benefit that comes with this method of training is robustness against image corruption. Due to its low computational cost and recent success, researchers have proposed many image mixing and deleting techniques. We furnish an in-depth survey of image mixing and deleting techniques and provide categorization via their most distinguishing features. We initiate our discussion with some fundamental relevant concepts. Next, we present essentials, such as each category’s strengths and limitations, describing their working mechanism, basic formulations, and applications. We also discuss the general challenges and recommend possible future research directions for image mixing and deleting data augmentation techniques. Datasets and codes for evaluation are publicly available here.

Review of image inpainting in practical challenge

As a very popular computer vision direction in recent years, image inpainting has developed very rapidly and produced many valuable results, but there are still many problems that need to be solved. This paper focuses on classic model research, related data sets, and evaluation standards. Through the analysis of specific examples, it is found that the effect of the existing model on the inpainting of some unique pictures is still unsatisfactory, and a solution is proposed to optimize the specificity of the training set by modifying the training set so that the model and the corrupt image are more correlated, which will result in more relative results.

Cross-modality Neuroimage Synthesis: A Survey

Multi-modality imaging improves disease diagnosis and reveals distinct deviations in tissues with anatomical properties. The existence of completely aligned and paired multi-modality neuroimaging data has proved its effectiveness in brain research. However, collecting fully aligned and paired data is expensive or even impractical, since it faces many difficulties, including high cost, long acquisition time, image corruption, and privacy issues. An alternative solution is to explore unsupervised or weakly supervised learning methods to synthesize the absent neuroimaging data. In this article, we provide a comprehensive review of cross-modality synthesis for neuroimages, from the perspectives of weakly supervised and unsupervised settings, loss functions, evaluation metrics, imaging modalities, datasets, and downstream applications based on synthesis. We begin by highlighting several opening challenges for cross-modality neuroimage synthesis. Then, we discuss representative architectures of cross-modality synthesis methods under different supervisions. This is followed by a stepwise in-depth analysis to evaluate how cross-modality neuroimage synthesis improves the performance of its downstream tasks. Finally, we summarize the existing research findings and point out future research directions. All resources are available at https://github.com/M-3LAB/awesome-multimodal-brain-image-systhesis.

End-to-end metric learning from corrupted images using triplet dimensionality reduction loss

In this paper, we present a novel dimensionality reduction term (i.e. TripletPCA and ContrastivePCA) designed to enhance the robustness of pair-based loss functions in metric learning against image corruptions. Notably, our approach achieves this without the need for any data preprocessing steps. Our method can be seamlessly integrated into existing models, such as a block with Plug & Play framework. To the best of our knowledge, our TripletPCA and ContrastivePCA represent the first attempts to incorporate dimensionality reduction directly into pair-based metric learning losses for end-to-end metric learning. By projecting image features into low-dimensional vectors, our proposed loss functions effectively retain the essential components of images while mitigating the impact of corrupted features. Consequently, our metric learning loss function accurately computes feature distances through the projection. Experimental results demonstrate that our dimensionality reduction term can be easily incorporated into various types of existing deep neural networks. This integration leads to a substantial improvement in performance on standard benchmark datasets for corrupted image classification tasks. On several corruption datasets, we achieve an average performance improvement of 10.55% compared to existing baseline methods. Our code is available at https://github.com/Juryun/TDRL.

ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors.

Image inpainting involves filling missing areas of a corrupted image. Despite impressive results have been achieved recently, restoring images with both vivid textures and reasonable structures remains a significant challenge. Previous methods have primarily addressed regular textures while disregarding holistic structures due to the limited receptive fields of Convolutional Neural Networks (CNNs). To this end, we study learning a Zero-initialized residual addition based Incremental Transformer on Structural priors (ZITS++), an improved model upon our conference work, ZITS (Dong et al. 2022). Specifically, given one corrupt image, we present the Transformer Structure Restorer (TSR) module to restore holistic structural priors at low image resolution, which are further upsampled by Simple Structure Upsampler (SSU) module to higher image resolution. To recover image texture details, we use the Fourier CNN Texture Restoration (FTR) module, which is strengthened by Fourier and large-kernel attention convolutions. Furthermore, to enhance the FTR, the upsampled structural priors from TSR are further processed by Structure Feature Encoder (SFE) and optimized with the Zero-initialized Residual Addition (ZeroRA) incrementally. Besides, a new masking positional encoding is proposed to encode the large irregular masks. Compared with ZITS, ZITS++ improves the FTR's stability and inpainting ability with several techniques. More importantly, we comprehensively explore the effects of various image priors for inpainting and investigate how to utilize them to address high-resolution image inpainting with extensive experiments. This investigation is orthogonal to most inpainting approaches and can thus significantly benefit the community.

Light Codes for Fast Two-Way Human-Centric Visual Communication

Visual codes, such as QR codes, are widely used in several applications for conveying information to users. However, user interactions based on spatial codes (e.g., displaying codes on phone screens for exchanging contact information) are often tedious, time consuming, and prone to errors due to image corruptions such as noise, blur, saturation, and perspective distortions. We propose Light Codes (LICO), a novel method for fast and fluid exchange of information among users. Light codes are based on transmitting and receiving temporal codes (instead of spatial) using compact and low-cost transceiver devices. The resulting approach enables seamless and near instantaneous exchange of short messages among users with minimal physical and cognitive effort. We design novel coding techniques, hardware prototypes, and applications that are optimized for human-centric communication, and facilitate fast and fluid user-to-user interactions in various challenging conditions, including a range of distances, motion, and ambient illumination. We evaluate the performance of the proposed methods both via quantitative analysis and user study based comparisons with several existing approaches including display-camera links, Bluetooth, and near-field communication, which show strong preference toward Light Codes in various real-world application scenarios.