Perceptual Fusion Research Articles

Bilinear Perceptual Fusion Algorithm Based on Brain Functional and Structural Data for ASD Diagnosis and Regions of Interest Identification.

Autism spectrum disorder (ASD) is a serious mental disorder with a complex pathogenesis mechanism and variable presentation among individuals. Although many deep learning algorithms have been used to diagnose ASD, most of them focus on a single modality of data, resulting in limited information extraction and poor stability. In this paper, we propose a bilinear perceptual fusion (BPF) algorithm that leverages data from multiple modalities. In our algorithm, different schemes are used to extract features according to the characteristics of functional and structural data. Through bilinear operations, the associations between the functional and structural features of each region of interest (ROI) are captured. Then the associations are used to integrate the feature representation. Graph convolutional neural networks (GCNs) can effectively utilize topology and node features in brain network analysis. Therefore, we design a deep learning framework called BPF-GCN and conduct experiments on publicly available ASD dataset. The results show that the classification accuracy of BPF-GCN reached 82.35%, surpassing existing methods. This demonstrates the superiority of its classification performance, and the framework can extract ROIs related to ASD. Our work provides a valuable reference for the timely diagnosis and treatment of ASD.

DNIM: Deep-sea netting intelligent enhancement and exposure monitoring using bio-vision

Intelligent monitoring of deep-sea nets is affected by light attenuation, light scattering, and limited dynamic range factors of the camera, which can cause color shift, low visibility, and over/underexposure in monitoring, resulting in reduced farming efficiency, and misdetection of biological behavior. Therefore, we propose a method for underwater net tank scene enhancement and exposure using multicolor space and bio-vision, called DNIM. Specifically, we first propose an intelligent enhancement and exposure monitoring method for deep-sea nets using bio-vision, which eliminates the effect of exposure on underwater imaging by introducing a bio-vision system that uses a brightness transformation function to generate a series of exposure images. Inspired by the image fusion method, we designed a color correction and visibility recovery method using bio-vision. In the Lab color model, color shifts in bio-vision are corrected by guiding feature transformations with contrast-constrained adaptive histogram equalization; in the RGB color model, depth information is recovered by creating a linear model for the depth of field of underwater blurred images based on the characteristics of the bio-vision system. To obtain more qualified results, we also propose an efficient perceptual fusion module to mix the output of visibility recovery and color correction. Finally, we propose two deep-sea net tank culture datasets: salmon bait-eating behavior dataset (SBBD) and salmon non-bait-eating behavior dataset (SNBD). By comparing the experimental analysis in four real scenario datasets, our DNIM achieved superior results compared to 16 advanced underwater enhancement methods. The code is publicly available at: https://github.com/An-Shunmin/DNIM.

Image fusion-based low-dose CBCT enhancement method for visualizing miniscrew insertion in the infrazygomatic crest

Digital dental technology covers oral cone-beam computed tomography (CBCT) image processing and low-dose CBCT dental applications. A low-dose CBCT image enhancement method based on image fusion is proposed to address the need for subzygomatic small screw insertion. Specifically, firstly, a sharpening correction module is proposed, where the CBCT image is sharpened to compensate for the loss of details in the underexposed/over-exposed region. Secondly, a visibility restoration module based on type II fuzzy sets is designed, and a contrast enhancement module using curve transformation is designed. In addition to this, we propose a perceptual fusion module that fuses visibility and contrast of oral CBCT images. As a result, the problems of overexposure/underexposure, low visibility, and low contrast that occur in oral CBCT images can be effectively addressed with consistent interpretability. The proposed algorithm was analyzed in comparison experiments with a variety of algorithms, as well as ablation experiments. After analysis, compared with advanced enhancement algorithms, this algorithm achieved excellent results in low-dose CBCT enhancement and effective observation of subzygomatic small screw implantation. Compared with the best performing method, the evaluation metric is 0.07–2 higher on both datasets. The project can be found at: https://github.com/sunpeipei2024/low-dose-CBCT.

MINM: Marine intelligent netting monitoring using multi-scattering model and multi-space transformation

Marine intelligent net tank aquaculture monitoring plays an important role in improving aquaculture efficiency, environmental monitoring efficiency, and environmental safety. The underwater environment has complex light, often with problems such as scattering and absorption, resulting in poor image quality, making it difficult to accurately analyze and judge the aquaculture environment. Improving marine intelligent net tank aquaculture monitoring has the following three advantages: 1) better observation and monitoring of the aquaculture process, timely detection of problems and abnormalities, to protect the benefits of aquaculture and product quality. 2) more convenient and rapid monitoring of the aquaculture environment, improving monitoring efficiency and reducing monitoring costs. 3) effective monitoring of the underwater environment around the farm, and timely detection of foreign pollution, harmful substances, and other problems, to protect the safety of the aquaculture environment. Therefore, in order to solve the two degradation problems of scattering and absorption in the process of marine smart net farm monitoring, we propose a marine smart net farm monitoring method using multiple scattering models and multiple spatial transformations, called MINM. Specifically, inspired by the image chromatic aberration correction method, we design a color correction method in the multicolor space, which is implemented by using the Lab and RGB color space by performing contrast-constrained adaptive histogram equalization and gray world assumptions, respectively, to correct color shifts in different color models. Based on this, we propose a de-scattering method using a multi-scattering model, which eliminates the effect of scattering on underwater imaging by embedding a complete multi-scattering underwater imaging model to guide the extraction of different features in the multi-scattering model. To obtain more qualified results, we also propose an efficient perceptual fusion to mix the output of the de-scattering and color correction. Thus, our method can take advantage of multiple scattering models and multiple spatial transformations to effectively improve the visual quality of underwater images, producing enhanced results that fit the complete underwater imaging model and have bio-visual characteristics. In extensive experimental demonstrations, our MINM method has shown higher performance than the state-of-the-art methods in terms of both visual quality and quantitative metrics. All experimental results and datasets in this paper are available from the following website: https://github.com/An-Shunmin/MINM.

Adaptive defogging method for transmission line inspection images based on multilayer perceptual fusion

Existing image defogging methods generally have problems such as incomplete defogging and color distortion. To address this problem, this paper proposes an adaptive defogging method for transmission line images based on multilayer perceptual fusion, which uses dynamic convolution, dense residuals, and attention mechanism to design an adaptive feature enhancement network containing six Dy-namic Residual Components (DRC) and two Dy-namic Skip-Connected Feature Fusion Component (DSCFF) composed of adaptive feature enhancement network, which prevents the problem of features being forgotten in the early stage of the network, and enhances the expressive ability of the model. For the decoding network, the de-fogging effect of the model is further strengthened by introducing a decoder module based on the SOS enhancement model, and finally, by comparing the experiments with the current de-fogging methods with more advanced performance, the results show that the method has good de-fogging effect and can retain the image details better with high color retention.

Consonance and Dissonance of Simultaneous Trichords in Western Music

Previous empirical studies have suggested that the perceived consonance/dissonance (C/D) of a musical chord depends on its psychoacoustic smoothness (lack of roughness), spectral harmonicity (perceptual fusion), and/or musical familiarity. We tested the dependence of C/D on smoothness and harmonicity in a hearing experiment that included all 19 possible trichords in musical pitch-class set theory. In each trial, a listener heard a chord (duration: 300 or 500 ms) and rated its C/D on an 11-point scale. Each trichord was presented 10 times: 4 times constructed from octave-complex tones (OCTs, sounding like an electronic organ) and 6 times from natural piano tones. Each OCT chord was presented in 4 different transpositions. The piano chords were in close or open position, and root position or 1st/2nd inversion (2 levels of spacing x 3 levels of inversion = 6 levels of voicing). We found no main effect of timbre (OCT versus piano) and no interaction between trichord and timbre. Results correlated closely with predictions of simple models of roughness and harmonicity. The roughness model performed better, and the predictions correlated with each other. A combined model was not superior to roughness alone. The results were consistent with a multifactorial model of the C/D of a musical chord, the main factors being roughness, harmonicity, and familiarity.

Underwater optical image enhancement based on super-resolution convolutional neural network and perceptual fusion.

Underwater optical images often have serious quality degradations and distortions, which hinders the development of underwater optics and vision systems. Currently, there are two mainstream solutions: non-learning based and learning-based. Both have their advantages and disadvantages. To fully integrate the advantages of both, we propose an enhancement method based on superresolution convolutional neural network (SRCNN) and perceptual fusion. First, we introduce a weighted fusion BL estimation model with a saturation correction factor (SCF-BLs fusion), the accuracy of image prior information is improved effectively. Next, a refined underwater dark channel prior (RUDCP) is proposed, which combines guided filtering and an adaptive reverse saturation map (ARSM) to restore the image, which not only preserves edge details but also avoids the interference of artificial light. Then, the SRCNN fusion adaptive contrast enhancement is proposed to enhance the colour and contrast. Finally, to further enhance image quality, we employ efficient perceptual fusion to blend the different resulting outputs. Extensive experiments demonstrate that our method has outstanding visual results in underwater optical image dehazing, color enhancement and is artefact- and halo-free.

AFT: Adaptive Fusion Transformer for Visible and Infrared Images.

In this paper, an Adaptive Fusion Transformer (AFT) is proposed for unsupervised pixel-level fusion of visible and infrared images. Different from the existing convolutional networks, transformer is adopted to model the relationship of multi-modality images and explore cross-modal interactions in AFT. The encoder of AFT uses a Multi-Head Self-attention (MSA) module and Feed Forward (FF) network for feature extraction. Then, a Multi-head Self-Fusion (MSF) module is designed for the adaptive perceptual fusion of the features. By sequentially stacking the MSF, MSA, and FF, a fusion decoder is constructed to gradually locate complementary features for recovering informative images. In addition, a structure-preserving loss is defined to enhance the visual quality of fused images. Extensive experiments are conducted on several datasets to compare our proposed AFT method with 21 popular approaches. The results show that AFT has state-of-the-art performance in both quantitative metrics and visual perception.

Triple-discriminator generative adversarial network for infrared and visible image fusion

We aim to address the challenging task of infrared and visible image fusion. The existed fusion methods cannot achieve the balance of clear boundaries and rich details. In this paper, we propose a novel fusion model using a triple-discriminator generative adversarial network, which can achieve the balance. The difference image obtained by image subtraction can highlight the difference information, extract image details, and obtain the target outlines in some scenes. Therefore, besides the visible discriminator and infrared discriminator, a new difference image discriminator is added to retain the difference between infrared and visible images, thereby improving the contrast of infrared targets and keeping the texture details in visible images. Multi-level features extracted by the discriminators are used for information measurement, and as a result, deriving perceptual fusion weights for adaptive fusion. SSIM loss function and target edge-enhancement loss are also introduced to improve the quality of the fused image. Compared with existing state-of-the-art fusion methods on public datasets, it is demonstrated that our model has a better performance on quantitative metrics and qualitative effects.

Perceptual Fusion of Electronic Chart and Marine Radar Image

Electronic charts and marine radars are indispensable equipment in ship navigation systems, and the fusion display of these two parts ensures that the vessel can display dangerous moving targets and various obstacles on the sea. To reduce the noise interference caused by external factors and hardware, a novel radar image denoising algorithm using the concept of Generative Adversarial Network (GAN) using Wasserstein distance is proposed. GAN focuses on transferring the image noise distribution between strong and weak noise, while the perceptual loss approach is to suppress the noise by comparing the perceptual characteristics of the output after denoising. Afterwards, an image registration method based on image transformation is proposed to eliminate the imaging difference between the radar image and chart image, in which the visual attribute transfer approach is used to transform images. Finally, the sparse theory is used to process the high frequency and low frequency subband coefficients of the detection image obtained by the fast Fourier transform in parallel to realizing the image fusion. The results show that the fused contour has a high consistency, fast training speed and short registration time.

Morphological Component Analysis-Based Perceptual Medical Image Fusion Using Convolutional Sparsity-Motivated PCNN

This paper proposes a perceptual medical image fusion framework based on morphological component analysis combining convolutional sparsity and pulse-coupled neural network, which is called MCA-CS-PCNN for short. Source images are first decomposed into cartoon components and texture components by morphological component analysis, and a convolutional sparse representation of cartoon layers and texture layers is produced by prelearned dictionaries. Then, convolutional sparsity is used as a stimulus to motivate the PCNN for dealing with cartoon layers and texture layers. Finally, the medical fused image is computed via combining fused cartoon layers and texture layers. Experimental results verify that the MCA-CS-PCNN model is superior to the state-of-the-art fusion strategy.

RefineDNet: A Weakly Supervised Refinement Framework for Single Image Dehazing

Haze-free images are the prerequisites of many vision systems and algorithms, and thus single image dehazing is of paramount importance in computer vision. In this field, prior-based methods have achieved initial success. However, they often introduce annoying artifacts to outputs because their priors can hardly fit all situations. By contrast, learning-based methods can generate more natural results. Nonetheless, due to the lack of paired foggy and clear outdoor images of the same scenes as training samples, their haze removal abilities are limited. In this work, we attempt to merge the merits of prior-based and learning-based approaches by dividing the dehazing task into two sub-tasks, i.e., visibility restoration and realness improvement. Specifically, we propose a two-stage weakly supervised dehazing framework, RefineDNet. In the first stage, RefineDNet adopts the dark channel prior to restore visibility. Then, in the second stage, it refines preliminary dehazing results of the first stage to improve realness via adversarial learning with unpaired foggy and clear images. To get more qualified results, we also propose an effective perceptual fusion strategy to blend different dehazing outputs. Extensive experiments corroborate that RefineDNet with the perceptual fusion has an outstanding haze removal capability and can also produce visually pleasing results. Even implemented with basic backbone networks, RefineDNet can outperform supervised dehazing approaches as well as other state-of-the-art methods on indoor and outdoor datasets. To make our results reproducible, relevant code and data are available at https://github.com/xiaofeng94/RefineDNet-for-dehazing.

Perceptual fusion of musical notes by native Amazonians suggests universal representations of musical intervals

Music perception is plausibly constrained by universal perceptual mechanisms adapted to natural sounds. Such constraints could arise from our dependence on harmonic frequency spectra for segregating concurrent sounds, but evidence has been circumstantial. We measured the extent to which concurrent musical notes are misperceived as a single sound, testing Westerners as well as native Amazonians with limited exposure to Western music. Both groups were more likely to mistake note combinations related by simple integer ratios as single sounds (‘fusion’). Thus, even with little exposure to Western harmony, acoustic constraints on sound segregation appear to induce perceptual structure on note combinations. However, fusion did not predict aesthetic judgments of intervals in Westerners, or in Amazonians, who were indifferent to consonance/dissonance. The results suggest universal perceptual mechanisms that could help explain cross-cultural regularities in musical systems, but indicate that these mechanisms interact with culture-specific influences to produce musical phenomena such as consonance.

A General Perceptual Infrared and Visible Image Fusion Framework Based on Linear Filter and Side Window Filtering Technology

Recently, edge-preserving filters have achieved great success in infrared (IR) and visible (VI) image fusion field. However, most edge-preserving filters are complex. In this paper, with the side window filtering technology by which most filters can improve their edge-preserving capabilities, we propose a general perceptual IR and VI image fusion framework with simple linear filter. Firstly, the source images are decomposed into edge feature components, hybrid components and base components by using linear filter and its side window version. Then, these components are combined by max-absolute fusion rule and improved max-absolute fusion rule. Finally, the fused image is reconstructed by adding all the fused components. In our experiments, two popular linear filters, i.e., box filter and Gaussian filter, are used to verify the effectiveness of the proposed framework. Experimental results show that the proposed fusion framework can obtain better perceptual fusion results than compared methods.

Rubber hands in space: the role of distance and relative position in the rubber hand illusion

The rubber hand illusion (RHI) is a perceptual phenomenon in which participants experience ownership over a fake model hand through synchronous visuotactile stimulation. Several studies have shown that the illusion occurs only when both hands are in close proximity to each other. In the present study, we systematically examined the role of relative position (lateral, distal) and distance (13–75 cm) of the model hand (with respect to participants’ real hand) on illusion experience across both lateral and distal positions. Furthermore, we also compared different facets of the subjective illusion experience; the experience of the model hand being part of one’s body (i.e., ownership) and the perceptual fusion of vision and touch (i.e., referral of touch). In two experiments we observed indications for a stronger illusion experiences in distal compared to lateral positions of identical distances, indicating that the illusory effects may vary as a function of the relative position of the hand. Our results also showed that manipulations of distance differently modulated both facets of the illusion. While ownership was restricted to near distances, referral of touch sensations remained stable at farther distances. These results are interpreted in relation to variations in sensory weighting across different planes.

Of Pipes and Patches: Listening to augmented pipe organs

Pipe organs are complex timbral synthesisers in an early acousmatic setting, which have always accompanied the evolution of music and technology. The most recent development is digital augmentation: the organ sound is captured, transformed and then played back in real time. The present augmented organ project relies on three main aesthetic principles: microphony, fusion and instrumentality. Microphony means that sounds are captured inside the organ case, close to the pipes. Real-time audio effects are then applied to the internal sounds before they are played back over loudspeakers; the transformed sounds interact with the original sounds of the pipe organ. The fusion principle exploits the blending effect of the acoustic space surrounding the instrument; the room response transforms the sounds of many single-sound sources into a consistent and organ-typical soundscape at the listener’s position. The instrumentality principle restricts electroacoustic processing to organ sounds only, excluding non-organ sound sources or samples. This article proposes a taxonomy of musical effects. It discusses aesthetic questions concerning the perceptual fusion of acoustic and electronic sources. Both extended playing techniques and digital audio can create musical gestures that conjoin the heterogeneous sonic worlds of pipe organs and electronics. This results in a paradoxical listening experience of unity in the diversity: the music is at the same time electroacoustic and instrumental.

Perceptual fusion of musical notes suggests universal representations of dissonance despite culture-dependent aesthetic associations

Music varies enormously across cultures, but some traits are widespread. Cross-cultural consistency in music could be driven by universal perceptual mechanisms adapted to natural sounds, but supporting evidence has been circumstantial due to the dearth of cross-cultural research. Here, we explore whether such perceptual mechanisms impose universal similarity relations on musical structure, potentially dissociating from culture-specific aesthetic judgments about music. We measured one possible signature of these similarity relations—the extent to which concurrent notes are perceived as a single sound—in members of a small-scale Amazonian society and Western listeners. We also measured aesthetic responses to the same stimuli. Unlike Westerners, Amazonian listeners were aesthetically indifferent to whether note combinations were canonically consonant (with aggregate frequency spectra resembling the harmonic series). However, Amazonians were nonetheless more likely to hear consonant combinations as a single sound, with fusion judgments that qualitatively resembled those of Western listeners. Thus, even in a culture with little exposure to Western harmony, reliance on harmonic frequency relations for sound segregation evidently induces consistent perceptual structure in note combinations. The results suggest that perceptual mechanisms for representing music can be shared across cultures, even though the perceptual equivalences that result give rise to culture-specific aesthetic associations.

The Integration of Occlusion and Disparity Information for Judging Depth in Autism Spectrum Disorder

In autism spectrum disorder (ASD), atypical integration of visual depth cues may be due to flattened perceptual priors or selective fusion. The current study attempts to disentangle these explanations by psychophysically assessing within-modality integration of ordinal (occlusion) and metric (disparity) depth cues while accounting for sensitivity to stereoscopic information. Participants included 22 individuals with ASD and 23 typically developing matched controls. Although adults with ASD were found to have significantly poorer stereoacuity, they were still able to automatically integrate conflicting depth cues, lending support to the idea that priors are intact in ASD. However, dissimilarities in response speed variability between the ASD and TD groups suggests that there may be differences in the perceptual decision-making aspect of the task.

Transsaccadic perceptual fusion.

Transsaccadic perceptual fusion is the integration of pre- and postsaccadic images into a single percept aligned in spatial coordinates. Several early studies reported an absence of transsaccadic fusion between dissimilar patterns, effectively stopping research on this question for three decades. We have now corrected two problematic aspects of these earlier studies and find robust evidence for transsaccadic perceptual fusion. First, we used simple pre- and postsaccadic targets, (|, ) for which spatial alignment is not critical. Second, we reduced the contrast of the postsaccadic stimulus, so that it would not suppress fusion. Participants reported seeing a superposition of the pre- and postsaccadic targets on 67% of trials. Importantly, we obtained similar results when the two stimuli were presented without an intervening eye movement, suggesting the existence of a general fusion mechanism. Directional biases in the saccade condition suggest that remapping might be the mechanism realigning the pre- and postsaccadic locations. Remapping may thus not only predict where targets will be located after a saccade but may also guide content, predicting what targets will look like. However, the constraints on the appearance of the fused percept suggest that it plays, at best, a limited role in visual stability across saccades.

A distortion-free contrast enhancement technique based on a perceptual fusion scheme

Contrast enhancement, in a broad sense, is a process whereby some characteristics of an image signal are highlighted. Techniques for image contrast enhancement improve the visibility of image details but may generate some undesirable artifacts such as noise amplification, ringing and overshooting. As a consequence, developing distortion-free methods for image enhancement is of great interest. In this paper, we propose a perceptual fusion technique to improve the performance of some existing contrast enhancement methods in terms of noise amplification. Multi-resolution fusion using the Laplacian pyramid decomposition is performed to account for the multi-channel characteristics of the human visual system (HVS). The results show the efficiency of the proposed method in enhancing details while preventing noise amplification.