Natural Images Research Articles

Discrepancies in perceived humanness between spatially filtered and unfiltered faces and their associations with uncanny feelings.

Human and artificial features that coexist in certain types of human-like robots create a discrepancy in perceived humanness and evoke uncanny feelings in human observers. However, whether this perceptual mismatch in humanness occurs for all faces, and whether it is related to the uncanny feelings toward them, is unknown. We investigated this by examining perceived humanness for a variety of natural images of robot and human faces with different spatial frequency (SF) information: that is, faces with only low SF, middle SF, and high SF information, and intact (spatially unfiltered) faces. Uncanny feelings elicited by these faces were also measured. The results showed perceptual mismatches that LSF, MSF, and HSF faces were perceived as more human than intact faces. This was particularly true for intact robot faces that looked slightly human, which tended to evoke strong uncanny feelings. Importantly, the mismatch in perceived humanness between the intact and spatially filtered faces was positively correlated with uncanny feelings toward intact faces. Given that the human visual system performs SF analysis when processing faces, the perceptual mismatches observed in this study likely occur in real life for all faces, and as such might be a ubiquitous source of uncanny feelings in real-life situations.

Seeing and Thinking about Urban Blue‒Green Space: Monitoring Public Landscape Preferences Using Bimodal Data

Urban blue‒green spaces (UBGSs) are a significant avenue for addressing the worldwide mental health crisis. To effectively optimise landscape design and management for the promotion of health benefits from UBGS, it is crucial to objectively understand public preferences. This paper proposes a method to evaluate public landscape preference from the perspective of seeing and thinking, takes the examples of seven parks around the Dianchi Lake in Kunming, China, and analyses the social media data by using natural language processing technology and image semantic segmentation technology. The conclusions are as follows: (1) The public exhibits significantly high positive sentiments towards various UBGSs, with over 93% of comments expressed positive sentiments. (2) Differences exist in the frequency and perception of landscape features between image and text modalities. Landscape elements related to stability are perceived more in images than in text, while dynamic and experiential elements are perceived more in text than in images. (3) In both modalities, the distinctive landscape features of parks are more frequently perceived and preferred by the public. In the end, the intrinsic links between landscape elements and public sentiment and preferences are discussed, and suggestions for design and management improvements are made to consolidate their health benefits to the public.

Nasal inhalation does not improve memory of visual repetitions.

Several studies suggest that breathing entrains neural oscillations and thereby improves visual detection and memory performance during nasal inhalation. However, the evidence for this association is mixed, with some studies finding no, minor, or opposite effects. Here, we tested whether nasal breathing phase influences memory of repeated images presented in a rapid serial visual presentation (RSVP) task. The RSVP task is ideal for studying the effects of respiratory-entrained oscillations on visual memory because it engages critical aspects of sensory encoding that depend on oscillatory activity, such as fast processing of natural images, repetition detection, memory encoding, and retrieval. It also enables the presentation of a large number of stimuli during each phase of the breathing cycle. In two separate experiments (n = 72 and n = 142, respectively) where participants were explicitly asked to breathe through their nose, we found that nasal breathing phase at target presentation did not significantly affect memory performance. An exploratory analysis in the first experiment suggested a potential benefit for targets appearing approximately 1 s after inhalation. However, this finding was not replicated in the pre-registered second experiment with a larger sample. Thus, in two large sample experiments, we found no measurable impact of breathing phase on memory performance in the RSVP task. These results suggest that the natural breathing cycle does not have a significant impact on memory for repeated images and raise doubts about the idea that visual memory is broadly affected by the breathing phase.

EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training.

The superior performance of modern computer vision backbones (e.g., vision Transformers learned on ImageNet-1K/22K) usually comes with a costly training procedure. This study contributes to this issue by generalizing the idea of curriculum learning beyond its original formulation, i.e., training models using easier-to-harder data. Specifically, we reformulate the training curriculum as a soft-selection function, which uncovers progressively more difficult patterns within each example during training, instead of performing easier-to-harder sample selection. Our work is inspired by an intriguing observation on the learning dynamics of visual backbones: during the earlier stages of training, the model predominantly learns to recognize some 'easier-to-learn' discriminative patterns in the data. These patterns, when observed through frequency and spatial domains, incorporate lower-frequency components, and the natural image contents without distortion or data augmentation. Motivated by these findings, we propose a curriculum where the model always leverages all the training data at every learning stage, yet the exposure to the 'easier-to-learn' patterns of each example is initiated first, with harder patterns gradually introduced as training progresses. To implement this idea in a computationally efficient way, we introduce a cropping operation in the Fourier spectrum of the inputs, enabling the model to learn from only the lower-frequency components. Then we show that exposing the contents of natural images can be readily achieved by modulating the intensity of data augmentation. Finally, we integrate these two aspects and design curriculum learning schedules by proposing tailored searching algorithms. Moreover, we present useful techniques for deploying our approach efficiently in challenging practical scenarios, such as large-scale parallel training, and limited input/output or data pre-processing speed. The resulting method, EfficientTrain++, is simple, general, yet surprisingly effective. As an off-the-shelf approach, it reduces the training time of various popular models (e.g., ResNet, ConvNeXt, DeiT, PVT, Swin, CSWin, and CAFormer) by [Formula: see text] on ImageNet-1K/22K without sacrificing accuracy. It also demonstrates efficacy in self-supervised learning (e.g., MAE). Code is available at: https://github.com/LeapLabTHU/EfficientTrain.

Fine-Grained Cross-Modal Semantic Consistency in Natural Conservation Image Data from a Multi-Task Perspective

Fine-grained representation is fundamental to species classification based on deep learning, and in this context, cross-modal contrastive learning is an effective method. The diversity of species coupled with the inherent contextual ambiguity of natural language poses a primary challenge in the cross-modal representation alignment of conservation area image data. Integrating cross-modal retrieval tasks with generation tasks contributes to cross-modal representation alignment based on contextual understanding. However, during the contrastive learning process, apart from learning the differences in the data itself, a pair of encoders inevitably learns the differences caused by encoder fluctuations. The latter leads to convergence shortcuts, resulting in poor representation quality and an inaccurate reflection of the similarity relationships between samples in the original dataset within the shared space of features. To achieve fine-grained cross-modal representation alignment, we first propose a residual attention network to enhance consistency during momentum updates in cross-modal encoders. Building upon this, we propose momentum encoding from a multi-task perspective as a bridge for cross-modal information, effectively improving cross-modal mutual information, representation quality, and optimizing the distribution of feature points within the cross-modal shared semantic space. By acquiring momentum encoding queues for cross-modal semantic understanding through multi-tasking, we align ambiguous natural language representations around the invariant image features of factual information, alleviating contextual ambiguity and enhancing model robustness. Experimental validation shows that our proposed multi-task perspective of cross-modal momentum encoders outperforms similar models on standardized image classification tasks and image–text cross-modal retrieval tasks on public datasets by up to 8% on the leaderboard, demonstrating the effectiveness of the proposed method. Qualitative experiments on our self-built conservation area image–text paired dataset show that our proposed method accurately performs cross-modal retrieval and generation tasks among 8142 species, proving its effectiveness on fine-grained cross-modal image–text conservation area image datasets.

AutoAugHAR: Automated Data Augmentation for Sensor-based Human Activity Recognition

Sensor-based HAR models face challenges in cross-subject generalization due to the complexities of data collection and annotation, impacting the size and representativeness of datasets. While data augmentation has been successfully employed in domains like natural language and image processing, its application in HAR remains underexplored. This study presents AutoAugHAR, an innovative two-stage gradient-based data augmentation optimization framework. AutoAugHAR is designed to take into account the unique attributes of candidate augmentation operations and the unique nature and challenges of HAR tasks. Notably, it optimizes the augmentation pipeline during HAR model training without substantially extending the training duration. In evaluations on eight inertial-measurement-units-based benchmark datasets using five HAR models, AutoAugHAR has demonstrated superior robustness and effectiveness compared to other leading data augmentation frameworks. A salient feature of AutoAugHAR is its model-agnostic design, allowing for its seamless integration with any HAR model without the need for structural modifications. Furthermore, we also demonstrate the generalizability and flexible extensibility of AutoAugHAR on four datasets from other adjacent domains. We strongly recommend its integration as a standard protocol in HAR model training and will release it as an open-source tool1.

Machine Learning-Based Predictive Models for Patients with Venous Thromboembolism: A Systematic Review.

Venous thromboembolism (VTE) is a chronic disorder with a significant health and economic burden. Several VTE-specific clinical prediction models (CPMs) have been used to assist physicians in decision-making but have several limitations. This systematic review explores if machine learning (ML) can enhance CPMs by analyzing extensive patient data derived from electronic health records. We aimed to explore ML-CPMs' applications in VTE for risk stratification, outcome prediction, diagnosis, and treatment. Three databases were searched: PubMed, Google Scholar, and IEEE electronic library. Inclusion criteria focused on studies using structured data, excluding non-English publications, studies on non-humans, and certain data types such as natural language processing and image processing. Studies involving pregnant women, cancer patients, and children were also excluded. After excluding irrelevant studies, a total of 77 studies were included. Most studies report that ML-CPMs outperformed traditional CPMs in terms of receiver operating area under the curve in the four clinical domains that were explored. However, the majority of the studies were retrospective, monocentric, and lacked detailed model architecture description and external validation, which are essential for quality audit. This review identified research gaps and highlighted challenges related to standardized reporting, reproducibility, and model comparison. ML-CPMs show promise in improving risk assessment and individualized treatment recommendations in VTE. Apparently, there is an urgent need for standardized reporting and methodology for ML models, external validation, prospective and real-world data studies, as well as interventional studies to evaluate the impact of artificial intelligence in VTE.

A binary neural computing unit with programmable gate using SFQ and CMOS hybrid circuit

Superconducting neural networks hold significant potential for future applications such as natural language processing and image recognition. To this end, we propose a binary neural computing unit implemented using a hybrid circuit of cryogenic CMOS and superconducting technologies. It offers two main advantages: firstly, we utilize current-mode computations for neural unit weight calculations, significantly reducing the unit’s footprint and enabling the potential for higher integration in the future. Secondly, all computations are performed in a low-temperature environment, which implies the possibility of on-chip learning in superconducting neural networks and the potential for achieving faster training rates in the future. We fabricated the chip using Nb 1 kA cm−2 process (1KP) technology and experimentally verified the correctness of the circuit logic. The margins for various control parameters of the circuit are approximately around 30%, and the superconducting circuit power consumption is estimated to be around 4 microwatts.

Robust prostate disease classification using transformers with discrete representations.

Automated prostate disease classification on multi-parametric MRI has recently shown promising results with the use of convolutional neural networks (CNNs). The vision transformer (ViT) is a convolutional free architecture which only exploits the self-attention mechanism and has surpassed CNNs in some natural imaging classification tasks. However, these models are not very robust to textural shifts in the input space. In MRI, we often have to deal with textural shift arising from varying acquisition protocols. Here, we focus on the ability of models to generalise well to new magnet strengths for MRI. We propose a new framework to improve the robustness of vision transformer-based models for disease classification by constructing discrete representations of the data using vector quantisation. We sample a subset of the discrete representations to form the input into a transformer-based model. We use cross-attention in our transformer model to combine the discrete representations of T2-weighted and apparent diffusion coefficient (ADC) images. We analyse the robustness of our model by training on a 1.5 T scanner and test on a 3 T scanner and vice versa. Our approach achieves SOTA performance for classification of lesions on prostate MRI and outperforms various other CNN and transformer-based models in terms of robustness to domain shift and perturbations in the input space. We develop a method to improve the robustness of transformer-based disease classification of prostate lesions on MRI using discrete representations of the T2-weighted and ADC images.

Exploring Generative Adversarial Networks for Face Generation

This research has used Generative Adversarial Networks (GANs) for face generation in the area of the research [1]. GANs have demonstrated a high effectiveness in imitation of natural images not only, but also in specific applications [2]. For the succeeding paper, their potential especially on creating human faces are being looked into. The primary goal is to develop a new design and practical training strategy to efficiently produce high-quality facial images [3]. Experimental results confirm the ability of the proposed method to generate artificially intelligent faces and its role in improving the quality of the images.

Enhanced Solar Coronal Imaging: A GAN Approach with Fused Attention and Perceptual Quality Enhancement

The activity of the solar corona has a significant impact on all aspects of human life. People typically use images obtained from astronomical telescopes to observe coronal activities, among which the Atmospheric Imaging Assembly (AIA) of the Solar Dynamics Observatory (SDO) is particularly widely used. However, due to resolution limitations, we have begun to study the application of generative adversarial network super-resolution techniques to enhance the image data quality for a clearer observation of the fine structures and dynamic processes in the solar atmosphere, which improves the prediction accuracy of solar activities. We aligned SDO/AIA images with images from the High-Resolution Coronal Imager (Hi-C) to create a dataset. This research proposes a new super-resolution method named SAFCSRGAN, which includes a spatial attention module that incorporates channel information, allowing the network model to better capture the corona’s features. A Charbonnier loss function was introduced to enhance the perceptual quality of the super-resolution images. Compared to the original method using ESRGAN, our method achieved an 11.9% increase in Peak Signal-to-Noise Ratio (PSNR) and a 4.8% increase in Structural Similarity (SSIM). Additionally, we introduced two perceptual image quality assessment metrics, the Natural Image Quality Evaluator (NIQE) and Learned Perceptual Image Patch Similarity (LPIPS), which improved perceptual quality by 10.8% and 1.3%, respectively. Finally, our experiments demonstrated that our improved model surpasses other models in restoring the details of coronal images.

Exploring age and gender variations in root canal morphology of maxillary premolars in Saudi sub population: a cross-sectional CBCT study

BackgroundIn complex teeth like maxillary premolars, endodontic treatment success depends on a complete comprehension of root canal anatomy. The research on mandibular premolars’ root canal anatomy has been extensive and well-documented in existing literature. However, there appears to be a notable gap in available data concerning the root canal anatomy of maxillary premolars. This study aimed to explore the root canal morphology of maxillary premolars using cone-beam computed tomography (CBCT) imaging, considering age and gender variations.MethodsFrom 500 patient CBCT scans, 787 maxillary premolar teeth were evaluated. The sample was divided by gender and age (10–20, 21–30, 31–40, 41–50, 51–60, and 61 years and older). Ahmed et al. classification system was used to record root canal morphology.ResultsThe most frequent classifications for right maxillary 1st premolars were 2MPM1 B1 L1 (39.03%) and 1MPM1 (2.81%), while the most frequent classifications for right maxillary 2nd premolars were 2MPM1 B1 L1 (39.08%) and 1MPM1 (17.85%). Most of the premolars typically had two roots (left maxillary first premolars: 81.5%, left maxillary second premolars: 82.7%, right maxillary first premolars: 74.4%, right maxillary second premolars: 75.7%). Left and right maxillary 1st premolars for classes 1MPM1 and 1MPM1–2−1 showed significant gender differences. For classifications 1MPM1 and 1MPM1–2−1, age-related changes were seen in the left and right maxillary first premolars.ConclusionThis study provides novel insights into the root canal anatomy of maxillary premolars within the Saudi population, addressing a notable gap in the literature specific to this demographic. Through CBCT imaging and analysis of large sample sizes, the complex and diverse nature of root canal morphology in these teeth among Saudi individuals is elucidated. The findings underscore the importance of CBCT imaging in precise treatment planning and decision-making tailored to the Saudi population. Consideration of age and gender-related variations further enhances understanding and aids in personalized endodontic interventions within this demographic.

Spectral Superresolution Using Transformer with Convolutional Spectral Self-Attention

Hyperspectral images (HSI) find extensive application across numerous domains of study. Spectral superresolution (SSR) refers to reconstructing HSIs from readily available RGB images using the mapping relationships between RGB images and HSIs. In recent years, convolutional neural networks (CNNs) have become widely adopted in SSR research, primarily because of their exceptional ability to extract features. However, most current CNN-based algorithms are weak in terms of extracting the spectral features of HSIs. While certain algorithms can reconstruct HSIs through the fusion of spectral and spatial data, their practical effectiveness is hindered by their substantial computational complexity. In light of these challenges, we propose a lightweight network, Transformer with convolutional spectral self-attention (TCSSA), for SSR. TCSSA comprises a CNN-Transformer encoder and a CNN-Transformer decoder, in which the convolutional spectral self-attention blocks (CSSABs) are the basic modules. Multiple cascaded encoding and decoding modules within TCSSA facilitate the efficient extraction of spatial and spectral contextual information from HSIs. The convolutional spectral self-attention (CSSA) as the basic unit of CSSAB combines CNN with self-attention in the transformer, effectively extracting both spatial local features and global spectral features from HSIs. Experimental validation of TCSSA’s effectiveness is performed on three distinct datasets: GF5 for remote sensing images along with CAVE and NTIRE2022 for natural images. The experimental results demonstrate that the proposed method achieves a harmonious balance between reconstruction performance and computational complexity.

A Prototype-Based Neural Network for Image Anomaly Detection and Localization

Image anomaly detection and localization perform not only image-level anomaly classification but also locate pixel-level anomaly regions. Recently, it has received much research attention due to its wide application in various fields. This paper proposes ProtoAD, a prototype-based neural network for image anomaly detection and localization. First, the patch features of normal images are extracted by a deep network pre-trained on nature images. Then, the prototypes of the normal patch features are learned by non-parametric clustering. Finally, we construct an image anomaly localization network (ProtoAD) by appending the feature extraction network with L2 feature normalization, a 1×1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1\ imes 1$$\\end{document} convolutional layer, a channel max-pooling, and a subtraction operation. We use the prototypes as the kernels of the 1×1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1\ imes 1$$\\end{document} convolutional layer; therefore, our neural network does not need a training phase and can conduct anomaly detection and localization in an end-to-end manner. Extensive experiments on two challenging industrial anomaly detection datasets, MVTec AD and BTAD, demonstrate that ProtoAD achieves competitive performance compared to the state-of-the-art methods with a higher inference speed. The code and pre-trained models are publicly available at https://github.com/98chao/ProtoAD.

TA-YOLO: a lightweight small object detection model based on multi-dimensional trans-attention module for remote sensing images

AbstractObject detection plays a vital role in remote sensing applications. Although object detection has achieved proud results in natural images, these methods are difficult to be directly applied to remote sensing images. Remote sensing images often have complex backgrounds and small objects, which results in a highly unbalanced distribution of foreground and complex background information. In order to solve the above problems, this paper proposes a multi-head channel and spatial trans-attention (MCSTA) module, which performs remote pixel interaction from the channel and spatial dimensions respectively to complete the attention feature capture function. It is a plug-and-play module that can be easily embedded in any other natural image object detection convolutional neural network, making it quickly applicable to remote sensing images. First, in order to reduce computational complexity and improve feature richness, we use a special linear convolution to obtain three projection features instead of the simple matrix multiplication transformation in Transformer. Second, we obtain trans-attention maps in different dimensions in a manner similar to the self-attention mechanism to capture the interrelationships of features in channels and spaces. In this process, we use a multi-head mechanism to perform parallel operations to improve speed. Furthermore, in order to avoid large-scale matrix operations, we specially designed an attention blocking mode to reduce computer memory usage and increase operation speed. Finally, we embedded the trans-attention module into YOLOv8, added a new detection head and optimized the feature fusion method, thus designing a lightweight small object detection model named TA-YOLO for remote sensing images. It has fewer parameters than the benchmark model YOLOv8, and its mAP on the PASCAL VOC and VisDrone data sets increased by 1.3% and 6.2% respectively. The experimental results prove the powerful function of the trans-attention module and the excellent performance of TA-YOLO.

Through their eyes: Multi-subject brain decoding with simple alignment techniques

Abstract To-date, brain decoding literature has focused on single-subject studies, that is, reconstructing stimuli presented to a subject under fMRI acquisition from the fMRI activity of the same subject. The objective of this study is to introduce a generalization technique that enables the decoding of a subject’s brain based on fMRI activity of another subject, that is, cross-subject brain decoding. To this end, we also explore cross-subject data alignment techniques. Data alignment is the attempt to register different subjects in a common anatomical or functional space for further and more general analysis. We utilized the Natural Scenes Dataset, a comprehensive 7T fMRI experiment focused on vision of natural images. The dataset contains fMRI data from multiple subjects exposed to 9,841 images, where 982 images have been viewed by all subjects. Our method involved training a decoding model on one subject’s data, aligning new data from other subjects to this space, and testing the decoding on the second subject based on information aligned to the first subject. We also compared different techniques for fMRI data alignment, specifically ridge regression, hyper alignment, and anatomical alignment. We found that cross-subject brain decoding is possible, even with a small subset of the dataset, specifically, using the common data, which are around 10% of the total data, namely 982 images, with performances in decoding comparable to the ones achieved by single-subject decoding. Cross-subject decoding is still feasible using half or a quarter of this number of images with slightly lower performances. Ridge regression emerged as the best method for functional alignment in fine-grained information decoding, outperforming all other techniques. By aligning multiple subjects, we achieved high-quality brain decoding and a potential reduction in scan time by 90%. This substantial decrease in scan time could open up unprecedented opportunities for more efficient experiment execution and further advancements in the field, which commonly requires prohibitive (20 hours) scan time per subject.

FAULT-TRANSFORMER: AN AUTOMATIC FAULT DETECTION ALGORITHM ON SEISMIC IMAGES USING A TRANSFORMER ENHANCED NEURAL NETWORK

Seismic fault detection is a key step in seismic interpretation and reservoir characterization that often requires a large amount of human labor and interpretation time. Therefore, automatic seismic fault detection is critical for improving the efficiency of seismic data processing and interpretation. Existing AI methods are mostly based on convolutional neural networks (CNNs) with a U-shaped encoder-decoder structure, known as the U-net. However, the convolution is limited in modeling long-range correlative features. Instead, Transformers, utilizing self-attention mechanisms, avoid the local nature of the convolution, which has the potential to extract long-distance correlations. Transformers are proven to perform well in natural language processing, image classification, and segmentation tasks in precision and recall. Here, we develop a new deep neural network with transformers and a U-net-like structure: Fault-Transformer, to perform the fault detection task. The new network outperforms traditional U-net in the application with synthetic data sets.

Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain.

A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.

Learning the Hybrid Nonlocal Self-Similarity Prior for Image Restoration

As an immensely important characteristic of natural images, the nonlocal self-similarity (NSS) prior has demonstrated great promise in a variety of inverse problems. Unfortunately, most current methods utilize either the internal or the external NSS prior learned from the degraded image or training images. The former is inevitably disturbed by degradation, while the latter is not adapted to the image to be restored. To mitigate such problems, this work proposes to learn a hybrid NSS prior from both internal images and external training images and employs it in image restoration tasks. To achieve our aims, we first learn internal and external NSS priors from the measured image and high-quality image sets, respectively. Then, with the learned priors, an efficient method, involving only singular value decomposition (SVD) and a simple weighting method, is developed to learn the HNSS prior for patch groups. Subsequently, taking the learned HNSS prior as the dictionary, we formulate a structural sparse representation model with adaptive regularization parameters called HNSS-SSR for image restoration, and a general and efficient image restoration algorithm is developed via an alternating minimization strategy. The experimental results indicate that the proposed HNSS-SSR-based restoration method exceeds many existing competition algorithms in PSNR and SSIM values.

GMIM: Self-supervised pre-training for 3D medical image segmentation with adaptive and hierarchical masked image modeling

Self-supervised pre-training and fully supervised fine-tuning paradigms have received much attention to solve the data annotation problem in deep learning fields. Compared with traditional pre-training on large natural image datasets, medical self-supervised learning methods learn rich representations derived from unlabeled data itself thus avoiding the distribution shift between different image domains. However, nowadays state-of-the-art medical pre-training methods were specifically designed for downstream tasks making them less flexible and difficult to apply to new tasks. In this paper, we propose grid mask image modeling, a flexible and general self-supervised method to pre-train medical vision transformers for 3D medical image segmentation. Our goal is to guide networks to learn the correlations between organs and tissues by reconstructing original images based on partial observations. The relationships are consistent within the human body and invariant to disease type or imaging modality. To achieve this, we design a Siamese framework consisting of an online branch and a target branch. An adaptive and hierarchical masking strategy is employed in the online branch to (1) learn the boundaries or small contextual mutation regions within images; (2) to learn high-level semantic representations from deeper layers of the multiscale encoder. In addition, the target branch provides representations for contrastive learning to further reduce representation redundancy. We evaluate our method through segmentation performance on two public datasets. The experimental results demonstrate our method outperforms other self-supervised methods. Codes are available at https://github.com/mobiletomb/Gmim.