Spatial Frequency Information Research Articles

Discrepancies in perceived humanness between spatially filtered and unfiltered faces and their associations with uncanny feelings.

Human and artificial features that coexist in certain types of human-like robots create a discrepancy in perceived humanness and evoke uncanny feelings in human observers. However, whether this perceptual mismatch in humanness occurs for all faces, and whether it is related to the uncanny feelings toward them, is unknown. We investigated this by examining perceived humanness for a variety of natural images of robot and human faces with different spatial frequency (SF) information: that is, faces with only low SF, middle SF, and high SF information, and intact (spatially unfiltered) faces. Uncanny feelings elicited by these faces were also measured. The results showed perceptual mismatches that LSF, MSF, and HSF faces were perceived as more human than intact faces. This was particularly true for intact robot faces that looked slightly human, which tended to evoke strong uncanny feelings. Importantly, the mismatch in perceived humanness between the intact and spatially filtered faces was positively correlated with uncanny feelings toward intact faces. Given that the human visual system performs SF analysis when processing faces, the perceptual mismatches observed in this study likely occur in real life for all faces, and as such might be a ubiquitous source of uncanny feelings in real-life situations.

On-machine frequency analysis of diamond turned surfaces with surface intrinsic mode decomposition

Evaluating surface frequency components in the fabrication process is critical for controlling the machined surface quality. The presence of anisotropic ripples on diamond-turned surfaces makes this challenging. A multiscale frequency evaluation method, referred to as Surface Intrinsic Mode Decomposition (SIMD), is proposed for evaluating on-machine surface measurement (OMSM) data. It decomposes continuous surface probing profiles, incorporating both temporal and spatial frequency information. In comparison to the conventional power spectral density (PSD) analysis method, the approach enriches frequency details over a wider range, which contributes to a more comprehensive understanding of surface quality and helps to identify mid-spatial frequency (MSF) errors.

The role of visual spatial frequencies in newborns' processing of dynamic facial expressions of emotion.

Evidence on newborns' discrimination of emotional facial expressions is scarce, and the question of what is the nature of the visual information that newborns rely on to perform such discrimination remains open. Here, we manipulated the spatial frequency (SF) content of the stimuli by selectively removing low spatial frequency (LSF) and high spatial frequency bands using newborn-appropriate cutoffs to investigate what information newborns use when preferring and discriminating between dynamic displays showing happy and fearful expressions unfolding over time. Using a preferential looking paradigm, in Study 1 (N = 63, 59% females, 92% White), we showed that newborns looked longer to happy over fearful expressions in unfiltered (broad spatial frequency) and high-pass (high spatial frequency > 0.6 cycles per degree [cpd]) faces but not in low-pass (LSF < 0.5 cpd) faces. In Study 2 (N = 22, 59% females, 91% White), newborns tested in a visual habituation paradigm showed successful discrimination of the two LSF emotions. Results show that newborns can discriminate between dynamic images of happy and fearful facial expressions containing either extreme low SF (< 0.5 cpd) information or higher SF (> 0.6 cpd) bandwidth. Their preference for the happy expression was present for intact and high-pass filtered faces but not for low-pass faces. This SF effect is tentatively driven by an enhancement of attentional response to the LSF fearful face, whereas the response to the happy face is unaffected by the SF manipulation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Focus-aware and deep restoration network with transformer for multi-focus image fusion

Multi-focus image fusion (MFIF) aims to break the limitations of physical devices and integrate different focused regions from multiple source images into a single fully focused image. However, existing fusion methods suffer from two issues when designing the fusion framework: over-reliance on post-processing operations or underestimation of their impact, leading to unsatisfactory visual results. To this end, we propose a fusion model that combines focus-aware and deep restoration. Specifically, we first employ a designed focus-aware module to integrate spatial frequency information from various levels of depth features in the source images. This integration produces an initial decision map that reflects the focus information. Then, we treat the optimization of the decision map as an image restoration task, rather than relying on commonly used post-processing operations in the MFIF field. In the restoration task, the initial decision map is passed as input along with the gradient information of the source images to the deep restoration network combined with the transformer for obtaining the final decision map. To facilitate the training of the reconstruction network, we also propose a large dataset with labels and a new set of loss functions. Extensive experimental results conclusively demonstrate that the proposed method produces fusion results of superior visual quality. This is substantiated by an assessment across eight objective metrics, which encompass normalized mutual information (1.193), nonlinear correlation information entropy (0.846), gradient-based similarity measurement (0.727), phase congruency-based measurements (0.844), structural similarity-based assessments (0.902), human perception-inspired evaluations (0.808), peak signal-to-noise ratio (74.55), and correlation coefficients (0.971). These metrics collectively affirm the outstanding fusion performance of our method. The code will be able to be found at https://github.com/govenda/FARfusion.

Improved modeling of human vision by incorporating robustness to blur in convolutional neural networks

Whenever a visual scene is cast onto the retina, much of it will appear degraded due to poor resolution in the periphery; moreover, optical defocus can cause blur in central vision. However, the pervasiveness of blurry or degraded input is typically overlooked in the training of convolutional neural networks (CNNs). We hypothesized that the absence of blurry training inputs may cause CNNs to rely excessively on high spatial frequency information for object recognition, thereby causing systematic deviations from biological vision. We evaluated this hypothesis by comparing standard CNNs with CNNs trained on a combination of clear and blurry images. We show that blur-trained CNNs outperform standard CNNs at predicting neural responses to objects across a variety of viewing conditions. Moreover, blur-trained CNNs acquire increased sensitivity to shape information and greater robustness to multiple forms of visual noise, leading to improved correspondence with human perception. Our results provide multi-faceted neurocomputational evidence that blurry visual experiences may be critical for conferring robustness to biological visual systems.

SFE-Net: Spatial-Frequency Enhancement Network for robust nuclei segmentation in histopathology images

Morphological features of individual nuclei serve as a dependable foundation for pathologists in making accurate diagnoses. Existing methods that rely on spatial information for feature extraction have achieved commendable results in nuclei segmentation tasks. However, these approaches are not sufficient to extract edge information of nuclei with small sizes and blurred outlines. Moreover, the lack of attention to the interior of the nuclei leads to significant internal inconsistencies. To address these challenges, we introduce a novel Spatial-Frequency Enhancement Network (SFE-Net) to incorporate spatial-frequency features and promote intra-nuclei consistency for robust nuclei segmentation. Specifically, SFE-Net incorporates a distinctive Spatial-Frequency Feature Extraction module and a Spatial-Guided Feature Enhancement module, which are designed to preserve spatial-frequency information and enhance feature representation respectively, to achieve comprehensive extraction of edge information. Furthermore, we introduce the Label-Guided Distillation method, which utilizes semantic features to guide the segmentation network in strengthening boundary constraints and learning the intra-nuclei consistency of individual nuclei, to improve the robustness of nuclei segmentation. Extensive experiments on three publicly available histopathology image datasets (MoNuSeg, TNBC and CryoNuSeg) demonstrate the superiority of our proposed method, which achieves 79.23%, 81.96% and 73.26% Aggregated Jaccard Index, respectively. The proposed model is available at https://github.com/jinshachen/SFE-Net.

Exploring Spatial Frequency Information for Enhanced Video Prediction Quality

Exploring Spatial Frequency Information for Enhanced Video Prediction Quality

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion.

multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

Removal of phase residues in electron holography.

Electron holography provides quantitative phase information regarding the electromagnetic fields and the morphology of micro- to nano-scale samples. A phase image reconstructed numerically from an electron hologram sometimes includes phase residues, i.e. origins of unremovable phase discontinuities, which make it much more difficult to quantitatively analyze local phase values. We developed a method to remove the residues in a phase image by a combination of patching local areas of a hologram and denoising based on machine learning. The small patches for a hologram, which were generated using the spatial frequency information of the own fringe patterns, were pasted at each residue point by an algorithm based on sparse modeling. After successive phase reconstruction, the phase components with no dependency on the vicinity were filtered out by Gaussian process regression. We determined that the phase discontinuities that appeared around phase residues were removed and the phase distributions of an atomic resolution phase image of a Pt nanoparticle were sufficiently restored.

Easily scalable multi-color DMD-based structured illumination microscopy.

Structured illumination microscopy (SIM) achieves super-resolution imaging using a series of phase-shifted sinusoidal illumination patterns to down-modulate high spatial-frequency information of samples. Digital micromirror devices (DMDs) have been increasingly used to generate SIM illumination patterns due to their high speed and moderate cost. However, a DMD micromirror array's blazed grating structure causes strong angular dispersion for different wavelengths of light, thus severely hampering its application in multicolor imaging. We developed a multi-color DMD-SIM setup that employs a diffraction grating to compensate the DMD's dispersion and demonstrate super-resolution SIM imaging of both fluorescent beads and live cells samples with four color channels. This simple but effective approach can be readily scaled to more color channels, thereby greatly expanding the application of SIM in the study of complex multi-component structures and dynamics in soft matter systems.

Express detection of visual objects by primate superior colliculus neurons

Primate superior colliculus (SC) neurons exhibit visual feature tuning properties and are implicated in a subcortical network hypothesized to mediate fast threat and/or conspecific detection. However, the mechanisms through which SC neurons contribute to peripheral object detection, for supporting rapid orienting responses, remain unclear. Here we explored whether, and how quickly, SC neurons detect real-life object stimuli. We presented experimentally-controlled gray-scale images of seven different object categories, and their corresponding luminance- and spectral-matched image controls, within the extrafoveal response fields of SC neurons. We found that all of our functionally-identified SC neuron types preferentially detected real-life objects even in their very first stimulus-evoked visual bursts. Intriguingly, even visually-responsive motor-related neurons exhibited such robust early object detection. We further identified spatial frequency information in visual images as an important, but not exhaustive, source for the earliest (within 100 ms) but not for the late (after 100 ms) component of object detection by SC neurons. Our results demonstrate rapid and robust detection of extrafoveal visual objects by the SC. Besides supporting recent evidence that even SC saccade-related motor bursts can preferentially represent visual objects, these results reveal a plausible mechanism through which rapid orienting responses to extrafoveal visual objects can be mediated.

Tandem aberration correction optics (TACO) in wide-field structured illumination microscopy.

Structured illumination microscopy (SIM) is a powerful super-resolution imaging technique that uses patterned illumination to down-modulate high spatial-frequency information of samples. However, the presence of spatially-dependent aberrations can severely disrupt the illumination pattern, limiting the quality of SIM imaging. Conventional adaptive optics (AO) techniques that employ wavefront correctors at the pupil plane are not capable of effectively correcting these spatially-dependent aberrations. We introduce the Tandem Aberration Correction Optics (TACO) approach that combines both pupil AO and conjugate AO for aberration correction in SIM. TACO incorporates a deformable mirror (DM) for pupil AO in the detection path to correct for global aberrations, while a spatial light modulator (SLM) is placed at the plane conjugate to the aberration source near the sample plane, termed conjugate AO, to compensate spatially-varying aberrations in the illumination path. Our numerical simulations and experimental results show that the TACO approach can recover the illumination pattern close to an ideal condition, even when severely misshaped by aberrations, resulting in high-quality super-resolution SIM reconstruction. The TACO approach resolves a critical traditional shortcoming of aberration correction for structured illumination. This advance significantly expands the application of SIM imaging in the study of complex, particularly biological, samples and should be effective in other wide-field microscopies.

Exploring a frequency-domain attention-guided cascade U-Net: Towards spatially tunable segmentation of vasculature

Developing fully automatic and highly accurate medical image segmentation methods is critically important for vascular disease diagnosis and treatment planning. Although advances in convolutional neural networks (CNNs) have spawned an array of automatic segmentation models converging to saturated high performance, none have explored whether CNNs can achieve (spatially) tunable segmentation. As a result, we propose multiple attention modules from a frequency-domain perspective to construct a unified CNN architecture for segmenting vasculature with desired (spatial) scales. The proposed CNN architecture is named frequency-domain attention-guided cascaded U-Net (FACU-Net). Specifically, FACU-Net contains two innovative components: (1) a frequency-domain-based channel attention module that adaptively tunes channel-wise feature responses and (2) a frequency-domain-based spatial attention module that enables the deep network to concentrate on foreground regions of interest (ROIs) effectively.Furthermore, we devised a novel frequency-domain-based content attention module to enhance or weaken the high (spatial) frequency information, allowing us to strengthen or eliminate vessels of interest. Extensive experiments using clinical data from patients with intracranial aneurysms (IA) and abdominal aortic aneurysms (AAA) demonstrated that the proposed FACU-Net met its design goal. In addition, we further investigated the association between varying (spatial) frequency components and the desirable vessel size/scale attributes. In summary, our preliminary findings are encouraging, and further developments may lead to deployable image segmentation models that are spatially tunable for clinical applications.

Imaging quality characterization and optimization of segmented planar imaging system based on honeycomb dense azimuth sampling lens array.

Segmented planar photoelectric imaging is an advanced computational imaging technology that utilizes photonic integrated circuits (PICs) to achieve the miniaturization of imaging systems. The original radial-spoke lens array has dense radial sampling and coarse azimuthal sampling. The sparsity and inhomogeneity of spatial frequency sampling lead to the loss of spatial frequency information and blurred reconstructed images. In this paper, a honeycomb dense azimuth sampling lens array is proposed, and three baseline pairing methods are designed, which can realize dense azimuth sampling, effectively increase spatial frequency sampling and improve the imaging quality. The signal transmission model of the segmented planar imaging system is established and the imaging process is simulated and analyzed. The simulation results show that the honeycomb lens array improves the azimuth sampling density and spatial frequency coverage, and its imaging quality is significantly improved compared with the hexagonal lens array and the radial-spoke lens array. Furthermore, the optimal choice of the baseline pairing method and the error range of the fill factor and are also given in this paper. The results indicate that the mixed pairing method first ensures low and medium-frequency dense sampling, and then increases high-frequency sampling, which makes the imaging results better than those of the other two baseline pairing methods in terms of image contour, contrast and image detail information. The sampling density of the spatial frequency and the imaging quality can be improved by increasing the fill factor. In the actual manufacturing process, the allowable error range of the fill factor of the lens array is within 5%. The research results will provide theoretical support for the design and development of segmented planar imaging system.

Robust multi-focus image fusion using focus property detection and deep image matting

Multi-focus image fusion techniques are designed to integrate the focused regions of multiple source images to produce an all-in-focus image. Unfortunately, the fusion results generated by existing methods are prone to suffer from chromatic aberrations and blurred boundary regions. In this paper, a multi-focus image fusion framework combining focus property detection and deep image matting is proposed to achieve high-quality fusion results. Specifically, we introduce spatial frequency detection blocks, which integrate the spatial frequency information of source images at different levels to achieve effective focus property detection for the initial decision. Then, the trimap acquired from the initial decision is optimized using a deep matting model so as to further detect the focused/defocused boundaries and obtain a more precise decision map. Meanwhile, a new hybrid loss function is introduced, and a large-scale dataset is created to improve the performance of the fusion framework. Finally, the fusion result is obtained by integrating the focused regions via the decision map. Compared to 18 other advanced fusion methods, the proposed method outperforms all of them in terms of the seven objective metrics evaluated on two publicly available datasets. These experimental results provide strong evidence that the proposed fusion framework excels not only in subjective vision but also in quantitative analysis. The code of the proposed method will be available at https://github.com/govenda/DIMF.

Spatial frequency enhanced salient object detection

Many salient object detection (SOD) methods based on convolutional neural networks utilize spatial frequency information from images to obtain salient features. However, most of these methods do not extract and/or exploit spatial frequency information effectively. This results in suboptimal detection performance in complex natural and human social scenes. In this work, we first propose an effective and flexible spatial frequency enhancement (SFE) module based on generalized Oct-convolution. SFE can effectively extract and incorporate multiple spatial frequency information from in-stage and cross-stage feature maps and output an arbitrary number of comprehensive and compact frequency features. Then we design a spatial frequency enhanced network (SFENet) adopting two SFE modules to refine high- and low-frequency salient features end-to-end. High-and low-frequency salient features are then integrated into a final full-band saliency prediction. Experiments show that SFENet can ensure the integrity and sharpness of salient regions in complex backgrounds on five benchmark datasets.

The effects of spatial frequency on the decoding of emotional facial expressions.

The accurate identification of emotion is critical to effectively navigating our social lives. However, it is not clear how distinct types of visual information afford the accurate perception of others' emotion states. Here, we sought to examine the influence of different spatial frequency visual information on emotion categorization, and whether distinctive emotional dimensions (valence and arousal) are differentially influenced by specific spatial frequency content. Across one pilot and two experiments (N = 603), we tested whether emotional facial expressions that vary in valence, arousal, and functional significance differ in accuracy of categorization as a function of low, intact, and high spatial frequency band information. Overall, we found a general decrease in the breadth of emotional expressions for filtered images but did not see a decrease in accuracy of categorization for the positive emotion, joy. Together, these results suggest that spatial frequency information influences perception of emotional expressions that differ in valence and arousal. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Holographic display using layered computer-generated volume hologram.

The spatial frequency of the reconstructed image of planar computer-generated hologram(CGH) is limited by the sampling interval and the lack of thickness. To break through this limitation of planar CGH, we propose a new computer-generated volume hologram(CGVH) for full-color dynamic holographic three-dimensional(3D) display, and an iteration-free layered CGVH generation method. The proposed CGVH is equivalent to a volume hologram sampled discretely in three directions. The generation method employs the layered angular spectral diffraction to calculate the light field in the layered CGVH, and then encodes it into a CGVH. Numerical simulation results show that the CGVH can accurately reconstruct full-color 3D objects, where better imaging quality, more concentrated diffraction energy, denser reconstructed spatial frequency information, and larger viewing angle are achieved. The proposed CGVH is expected to be applied to realize dynamic modulation, wavelength multiplexing, and angle multiplexing in various optical fields in the future.

Cascaded Convolutional Recurrent Neural Networks for EEG Emotion Recognition Based on Temporal–Frequency–Spatial Features

Emotion recognition is a research area that spans multiple disciplines, including computational science, neuroscience, and cognitive psychology. The use of electroencephalogram (EEG) signals in emotion recognition is particularly promising due to their objective and nonartefactual nature. To effectively leverage the spatial information between electrodes, the temporal correlation of EEG sequences, and the various sub-bands of information corresponding to different emotions, we construct a 4D matrix comprising temporal–frequency–spatial features as the input to our proposed hybrid model. This model incorporates a residual network based on depthwise convolution (DC) and pointwise convolution (PC), which not only extracts the spatial–frequency information in the input signal, but also reduces the training parameters. To further improve performance, we apply frequency channel attention networks (FcaNet) to distribute weights to different channel features. Finally, we use a bidirectional long short-term memory network (Bi-LSTM) to learn the temporal information in the sequence in both directions. To highlight the temporal importance of the frame window in the sample, we choose the weighted sum of the hidden layer states at all frame moments as the input to softmax. Our experimental results demonstrate that the proposed method achieves excellent recognition performance. We experimentally validated all proposed methods on the DEAP dataset, which has authoritative status in the EEG emotion recognition domain. The average accuracy achieved was 97.84% for the four binary classifications of valence, arousal, dominance, and liking and 88.46% for the four classifications of high and low valence–arousal recognition.

Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

The rise of automation and machine learning (ML) in electron microscopy has the potential to revolutionize materials research through autonomous data collection and processing. A significant challenge lies in developing ML models that rapidly generalize to large data sets under varying experimental conditions. We address this by employing a cycle generative adversarial network (CycleGAN) with a reciprocal space discriminator, which augments simulated data with realistic spatial frequency information. This allows the CycleGAN to generate images nearly indistinguishable from real data and provide labels for ML applications. We showcase our approach by training a fully convolutional network (FCN) to identify single atom defects in a 4.5 million atom data set, collected using automated acquisition in an aberration-corrected scanning transmission electron microscope (STEM). Our method produces adaptable FCNs that can adjust to dynamically changing experimental variables with minimal intervention, marking a crucial step towards fully autonomous harnessing of microscopy big data.