Human Perceptual System Research Articles

Bidirectional Photovoltage-Driven Oxide Transistors for Neuromorphic Visual Sensors.

Biological vision is one of the most important parts of the human perception system. However, emulating biological visuals is challenging because it requires complementary photoexcitation and photoinhibition. Here, the study presents a bidirectional photovoltage-driven neuromorphic visual sensor (BPNVS) that is constructed by monolithically integrating two perovskite solar cells (PSCs) with dual-gate ion-gel-gated oxide transistors. PSCs act as photoreceptors, converting external visual stimuli into electrical signals, whereas oxide transistors generate neuromorphic signal outputs that can be adjusted to produce positive and negative photoresponses. This device mimics the human vision system's ability to recognize colored and color-mixed patterns. The device achieves a static color recognition accuracy of 96% by utilizing the reservoir computing system for feature extraction. The BPNVS mem-reservoir chip is also proposed for handing object movement and dynamic color recognition. This work is a significant step forward in neuromorphic sensing and complex pattern recognition.

A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social Environments

Human–robot interaction (HRI) in human social environments (HSEs) poses unique challenges for robot perception systems, which must combine asynchronous, heterogeneous data streams in real time. Multimodal perception systems are well-suited for HRI in HSEs and can provide more rich, robust interaction for robots operating among humans. In this article, we provide an overview of multimodal perception systems being used in HSEs, which is intended to be an introduction to the topic and summary of relevant trends, techniques, resources, challenges, and terminology. We surveyed 15 peer-reviewed robotics and HRI publications over the past 10+ years, providing details about the data acquisition, processing, and fusion techniques used in 65 multimodal perception systems across various HRI domains. Our survey provides information about hardware, software, datasets, and methods currently available for HRI perception research, as well as how these perception systems are being applied in HSEs. Based on the survey, we summarize trends, challenges, and limitations of multimodal human perception systems for robots, then identify resources for researchers and developers and propose future research areas to advance the field.

What Is the Work of Animation? The Plasticity of Time in the Fourth-Dimensional Image

Plasticity is a central concept in the philosophy of Catherine Malabou and the theory of animation. And yet, the temporal dimension of transformation and mutability that Malabou foregrounds in her philosophy is missing from recent theoretical accounts of plasticity in animation studies​. ​In these accounts, plasticity appears as a mode of spatial extension, conveying the elasticity of form across a Cartesian timeline, whereas Malabou aims to understand time’s own metamorphosis and self-differentiation. This article invokes Malabou’s theorization of plasticity as the becoming of time itself rather than the becoming of things in time to rethink the perception of life-like movement in animation. Reviewing the techniques of animation that Frank Thomas and Ollie Johnston describes in The Illusion of Life as “drawing in the fourth dimension”, I argue that Malabou’s characterization of plasticity as the capacity “to see (what is) coming” ( voir venir) gets to the heart of animation as an art of subjective anticipation. Utilizing the human perceptual system’s tendency to invent new structures and relationships in response to changing inputs from the environment, animation prepares us to anticipate the unforeseeable and to perceive an order of movement typically concealed by the movement of objects in space.

Unveiling Gating Behavior in Piezoionic Effect: toward Neuromimetic Tactile Sensing.

The human perception system's information processing is intricately linked to the nonlinear response and gating effect of neurons. While piezoionics holds potential in emulating the pressure sensing capability of biological skin, the incorporation of information processing functions seems neglected. Here, ionic gating behavior in piezoionic hydrogels is uncovered as a notable extension beyond the previously observed linear responses. The hydrogel can generate remarkably high voltages (700mV) and currents (7mA) when indentation forces surpass the threshold. Through a comprehensive analysis involving simulations and experimental investigations, it is proposed that the gating behavior emerges due to significant diffusion differences between cations and anions. To showcase the practical implications of this breakthrough, the piezoionic hydrogels are successfully integrated with prostheses and robot hands, demonstrating that the gating effect enables accurate discrimination between gentle and harsh touch. The advancement in neuromimetic tactile sensing has significant potential for emerging applications such as humanoid robotics and biomedical engineering, offering valuable opportunities for further development of embodied neuromorphic intelligence.

Colour/shape-taste correspondences across three languages in ChatGPT

Crossmodal correspondences, the tendency for a sensory feature / attribute in one sensory modality (either physically present or merely imagined), to be associated with a sensory feature in another sensory modality, have been studied extensively, revealing consistent patterns, such as sweet tastes being associated with pink colours and round shapes across languages. The present research explores whether such correspondences are captured by ChatGPT, a large language model developed by OpenAI. Across twelve studies, this research investigates colour/shapes-taste crossmodal correspondences in ChatGPT-3.5 and -4o, focusing on associations between shapes/colours and the five basic tastes across three languages (English, Japanese, and Spanish). Studies 1A-F examined taste-shape associations, using prompts in three languages to assess ChatGPT's association of round and angular shapes with the five basic tastes. The results indicated significant, consistent, associations between shape and taste, with, for example, round shapes strongly associated with sweet/umami tastes and angular shapes with bitter/salty/sour tastes. The magnitude of shape-taste matching appears to be greater in ChatGPT-4o than in ChatGPT-3.5, and ChatGPT prompted in English and Spanish than ChatGPT prompted in Japanese. Studies 2A-F focused on colour-taste correspondences, using ChatGPT to assess associations between eleven colours and the five basic tastes. The results indicated that ChatGPT-4o, but not ChatGPT-3.5, generally replicates the patterns of colour-taste correspondences that have previously been observed in human participants. Specifically, ChatGPT-4o associates sweet tastes with pink, sour with yellow, salty with white/blue, bitter with black, and umami with red across languages. However, the magnitude/similarity of shape/colour-taste matching observed in ChatGPT-4o appears to be more pronounced (i.e., having little variance, large mean difference), which does not adequately reflect the subtle nuances typically seen in human shape/colour-taste correspondences. These findings suggest that ChatGPT captures colour/shapes-taste correspondences, with language- and GPT version-specific variations, albeit with some differences when compared to previous studies involving human participants. These findings contribute valuable knowledge to the field of crossmodal correspondences, explore the possibility of generative AI that resembles human perceptual systems and cognition across languages, and provide insight into the development and evolution of generative AI systems that capture human crossmodal correspondences.

Light-Adapted Optoelectronic-Memristive Device for the Artificial Visual System.

With the development of artificial intelligence systems, it is necessary to develop optoelectronic devices with photoresponse and storage capacity to simulate human visual perception systems. The key to an artificial visual perception system is to integrate components with both sensing and storage capabilities of illumination information. Although module integration components have made useful progress, they still face challenges such as multispectral response and high energy consumption. Here, we developed a light-adapted optoelectronic-memristive device integrated by an organic photodetector and ferroelectric-based memristor to simulate human visual perception. ITO/P3HT:PC71BM/Au as the light sensor unit shows a high on/off ratio (Iph/Id) reaching ∼5 × 104 at 0 V. The memristor unit, consisting of ITO/CBI@P(VDF-TrFE)/Cu, has a RON/ROFF ratio window of ∼106 under 0.05 V read voltage and ultralow power consumption of ∼1 pW. Moreover, the artificial visual perception unit shows stable light-adapted memory windows under different wavelengths of irradiation light (400, 500, and 600 nm; they meet the spectral range of human visual recognition) and can clearly identify the target image ("T" shape) because of the apparent contrast, which results from the high ROFF/RON ratio values. These results provide a potential design strategy for the development of intelligent artificial vision systems.

Action-guided prompt tuning for video grounding

Video grounding aims to locate a moment-of-interest semantically corresponding to a given query. We claim that existing methods overlook two critical issues: (1) the sparsity of language, and (2) the human perception process of events. To be specific, previous studies forcibly map the video modality and language modality into a joint space for alignment, disregarding their inherent disparities. Verbs play a crucial role in queries, providing discriminative information for distinguishing different videos. However, in the video modality, actions especially salient ones, are typically manifested through a greater number of frames, encompassing a richer reservoir of informative details. At the query level, verbs are constrained to a single word representation,creating a disparity. This discrepancy highlights a significant sparsity in language features, resulting in the suboptimality of mapping the two modalities into a shared space naively. Furthermore, segmenting ongoing activity into meaningful events is integral to human perception and contributes to event memory. Preceding methods fail to account for this essential perception process. Considering the aforementioned issues, we propose a novel Action-Guided Prompt Tuning (AGPT) method for video grounding. Firstly, we design a Prompt Exploration module to explore latent expansion information of salient verbs in language,thereby reducing the language feature sparsity and facilitating cross-modal matching. Secondly, we design the auxiliary task of action temporal prediction for video grounding and introduce a temporal rank loss function to simulate the human perceptual system’s segmentation of events, rendering our AGPT to be temporal-aware. Our approach can be seamlessly integrated into any video grounding model with minimal additional parameters. Extensive ablation experiments on three backbones and three datasets demonstrate the superiority of our method.

Biological Basis and Computer Vision Applications of Image Phase Congruency: A Comprehensive Survey.

The concept of Image Phase Congruency (IPC) is deeply rooted in the way the human visual system interprets and processes spatial frequency information. It plays an important role in visual perception, influencing our capacity to identify objects, recognize textures, and decipher spatial relationships in our environments. IPC is robust to changes in lighting, contrast, and other variables that might modify the amplitude of light waves yet leave their relative phase unchanged. This characteristic is vital for perceptual tasks as it ensures the consistent detection of features regardless of fluctuations in illumination or other environmental factors. It can also impact cognitive and emotional responses; cohesive phase information across elements fosters a perception of unity or harmony, while inconsistencies can engender a sense of discord or tension. In this survey, we begin by examining the evidence from biological vision studies suggesting that IPC is employed by the human perceptual system. We proceed to outline the typical mathematical representation and different computational approaches to IPC. We then summarize the extensive applications of IPC in computer vision, including denoise, image quality assessment, feature detection and description, image segmentation, image registration, image fusion, and object detection, among other uses, and illustrate its advantages with a number of examples. Finally, we discuss the current challenges associated with the practical applications of IPC and potential avenues for enhancement.

Auditory localization: a comprehensive practical review.

Auditory localization is a fundamental ability that allows to perceive the spatial location of a sound source in the environment. The present work aims to provide a comprehensive overview of the mechanisms and acoustic cues used by the human perceptual system to achieve such accurate auditory localization. Acoustic cues are derived from the physical properties of sound waves, and many factors allow and influence auditory localization abilities. This review presents the monaural and binaural perceptual mechanisms involved in auditory localization in the three dimensions. Besides the main mechanisms of Interaural Time Difference, Interaural Level Difference and Head Related Transfer Function, secondary important elements such as reverberation and motion, are also analyzed. For each mechanism, the perceptual limits of localization abilities are presented. A section is specifically devoted to reference systems in space, and to the pointing methods used in experimental research. Finally, some cases of misperception and auditory illusion are described. More than a simple description of the perceptual mechanisms underlying localization, this paper is intended to provide also practical information available for experiments and work in the auditory field.

Topological hydrogen-bonded organic frameworks (HOFs) and their electronic applications in sensor, memristor, and neuromorphic computing

Recently, an emerging class of hydrogen-bonded organic frameworks (HOFs) has become an appealing member of organic material family, attributed to their layered self-assembly structures, high-crystalline, and environmentally friendly characteristics, which have rapidly propelled their development in the field of electronic devices. In this context, we focus on the latest category of topological HOFs, with particular attention given to cutting-edge experimental demonstrations, exceptional electrical performances, and promising applications. First, a concise concept and fundamental mechanism of HOFs are provided, elucidating the potential correlation between structural designs and material properties. Subsequently, a comprehensive summary is presented on the preparation and synthesis methods, such as hydrothermal techniques, epitaxial growth, electro-deposition, among others. Notably, the latest advancements in HOFs-based electronics are thoroughly introduced and discussed, along with their applications in sensors, memristors, artificial synapses, neuromorphic computing, and human perception systems. Finally, the future challenges and prospects of topological HOFs are elaborated upon with the aim of providing valuable guidance for high-performance HOF-based electronics.

Perceptual Narrowing in Face Processing: Reviewing the Factors Influencing its Onset and Offset

AbstractAn infant's face‐ and speech‐processing system develops during the first year from broad and non‐specific to becoming a system that is tuned to the faces and languages to which they are most exposed. This phenomenon is called perceptual narrowing. Before 9 months of age, infants are capable of discriminating and recognizing individuals from any type of race/species faces. However, with increased exposure to own‐race and own‐species faces and lack of exposure to other types of faces, by 9 months of age, they have improved their ability to discriminate own‐race faces, while they show increased difficulty in the discrimination of faces from other races and species. According to the literature, we can conclude that, by 12 months of age, most human perceptual systems have become perceptually tuned and adult‐like; however, this is not true. In the following sections, we will argue that perceptual narrowing for faces occurs during the same developmental period as it does for language, and that it can be prevented or modulated with sufficient exposure to unfamiliar sounds or face types. We conclude that narrowing has been designed by natural selection to tailor an individual's cognition to their local social context. It might occur for every domain pertaining to social communication—from speech processing to emotion or gesture perception—in a process that gradually adapts the infant to their native social group.

Bionic visual-audio photodetectors with in-sensor perception and preprocessing.

Serving as the "eyes" and "ears" of the Internet of Things, optical and acoustic sensors are the fundamental components in hardware systems. Nowadays, mainstream hardware systems, often comprising numerous discrete sensors, conversion modules, and processing units, tend to result in complex architectures that are less efficient compared to human sensory pathways. Here, a visual-audio photodetector inspired by the human perception system is proposed to enable all-in-one visual and acoustic signal detection with computing capability. This device not only captures light but also optically records sound waves, thus achieving "watching" and "listening" within a single unit. The gate-tunable positive, negative, and zero photoresponses lead to highly programmable responsivities. This programmability enables the execution of diverse functions, including visual feature extraction, object classification, and sound wave manipulation. These results showcase the potential of expanding perception approaches in neuromorphic devices, opening up new possibilities to craft intelligent and compact hardware systems.

Refocus the Attention for Parameter-Efficient Thermal Infrared Object Tracking.

Introducing deep trackers to thermal infrared (TIR) tracking is hampered by the scarcity of large training datasets. To alleviate the predicament, a common approach is full fine-tuning (FFT) based on pretrained RGB parameters. Nevertheless, due to its inefficient training pattern and representation collapse risk, some parameter-efficient fine-tuning (PEFT) alternatives have been promoted recently. However, the existing PEFT algorithms typically follow a bottom-up way, where their attention solely relies on the input and lacks the capability of task-guided top-down attention, which provides the task-relevant representation such as the human visual perception system. In this article, we introduce ReFocus, a new PEFT method that adapts the pretrained RGB foundation tracking model to the downstream TIR tracking task through the guidance of high-level task-specific signals in a top-down attention manner. By freezing the entire foundation model and only training query-guided feature selection and top-down blocks, ReFocus achieves state-of-the-art (SOTA) TIR tracking performance while keeping training efficiency. Extensive experiments on five TIR tracking benchmarks demonstrate that ReFocus significantly improves the performance of the foundation tracker. Besides, further ablation studies show the effectiveness and flexible adaptability of the proposed method to lighter foundation models and different tracking frameworks. Compared to FFT and other bottom-up PEFT paradigms, such as head probe, low-rank adaptation (LoRA), and adapter, our method achieves comparable or superior performance with fewer training parameters and reveals the advantage of learning stability.

Polyphonic Sound Event Detection Using Mel-Pseudo Constant Q-Transform and Deep Neural Network

The task of identification of sound events in a particular surrounding is known as Sound Event Detection (SED) or Acoustic Event Detection (AED). The occurrence of sound events is unstructured and also displays wide variations in both temporal structure and frequency content. Sound events may be non-overlapped (monophonic) or overlapped (polyphonic) in nature. In real-time scenarios, polyphonic SED is most commonly seen as compared to monophonic SED. In this paper, a Mel-Pseudo Constant Q-Transform (MP-CQT) technique is introduced to perform polyphonic SED to effectively learn both monophonic and polyphonic sound events. A pseudo CQT technique is adapted to extract features from the audio files and their Mel spectrograms. The Mel-scale is believed to broadly simulate human perception system. The classifier used is a Convolutional Recurrent Neural Network (CRNN). Comparison of the performance of the proposed MP-CQT technique along with CRNN is presented and a considerable performance improvement is observed. The proposed method achieved an average error rate of 0.684 and average F1 score of 52.3%. The proposed approach is also analyzed for the robustness by adding an additional noise at different Signal to Noise Ratios (SNRs) to the audio files. The proposed method for SED task has displayed improved performance as compared to state-of-the-art SED systems. The introduction of new feature extraction technique has shown promising improvement in the performance of the polyphonic SED system.

Multilevel Feature Fusion for End-to-End Blind Image Quality Assessment

In this paper, a framework based on two feature extraction networks and a multilevel feature fusion (MFF) network is proposed. Multilevel degradation features can be obtained through this method, and combined with the human visual perception system, the local and global feature information contained in these features can be captured, which is conducive to the prediction of distorted images. First, a restored image approximating a reference image is generated by a restorative generative adversarial network (GAN). Furthermore, the multilevel degradation features of distorted images and the restored image features are extracted by EfficientNet. Second, the features extracted by EfficientNet are input into the MFF network and are fully expressed by the top-down, bottom-up and third edge joining methods. Moreover, the features provide more low-level details and high-level semantic features for the prediction of image quality scores. In addition, after the MFF stage, the framework calculates the score of each branch feature and obtains the average quality score. Experimental results show that our method achieves greatly improved prediction accuracy and performance on five standard databases.

ICONICITY IN THE BINOMIALS OF THE GLORIOUS QURAN: AL-BAQARAH SURA AS A CASE STUDY

This study investigates evidence for the existence of iconicity as a processing strategy in Quranic discourse. It is argued here that the Muslim’s most glorious book, the Quran, is an excellent source of morphosyntactic iconicity in the Arabic language. The hypothesis was tested by analysing sixty-six binomials from Al-Baqarah Sura within a synthetic model based on an extension of Givón’s 1984 model of iconicity. The findings show that some of the sampled iconic binomials accord with these principles, while others are iconic with regard to the human perceptual system. This result challenges Saussurian arbitrariness and lends support for the view that some aspects of language are iconic. Furthermore, iconic binomials are richly manifested in Quranic lexicon and discourse. This is because they are part of the rhetoric of Quran. Allah has created everything in pairs: night and day, death and life, sun and moon, paradise and hell, etc. These pairs express such rhetorical functions as revealing situation, warning, promising, inhibition, specification, etc. They also express Allah’s intention in motivating people to make a balance between benefits and problems of each pair. The role played by iconic binomials can be easily tested by removing some of them from the suras of Quran and asking what happens then. On such cases, the Quranic rhetoric is distorted rather than unites. Above all, such binomials are cross linguistics, they are universal since they are found in all languages. This conclusions challenges Sapir-Worf hypothesis and lends support to an assumption of a minimal universality of linguistic interactions.

Robust PDF Watermarking against Print-Scan Attack.

Portable document format (PDF) files are widely used in file transmission, exchange, and circulation because of their platform independence, small size, good browsing quality, and the ability to place hyperlinks. However, their security issues are also more thorny. It is common to distribute printed PDF files to different groups and individuals after printing. However, most PDF watermarking algorithms currently cannot resist print-scan attacks, making it difficult to apply them in leak tracing of both paper and scanned versions of PDF documents. To tackle this issue, we propose an invisible digital watermarking technology based on modifying the edge pixels of text strokes to hide information in PDFs, which achieves high robustness to print-scan attacks. Moreover, it cannot be detected by human perception systems. This method focuses on the representation of text by embedding watermarks by changing the features of the text to ensure that changes in these features can be reflected in the scanned PDF after printing. We first segment each text line into two sub-blocks, then select the row of pixels with the most black pixels, and flip the edge pixels closest to this row. This method requires the participation of original PDF documents in detection. The experimental results show that all peak signal-to-noise ratio (PSNR) values of our proposed method exceed 32 dB, which indicates satisfactory invisibility. Meanwhile, this method can extract the hidden information with 100% accuracy under the JPEG compression attack, and has high robustness against noise attacks and print-scan attacks. In the case of no attacks, the watermark can be recovered without any loss. In terms of practical applications, our method can be applied in the practical leak tracing of official paper documents after distribution.

A Remote Sensing Image Quality Interpretation Scale Characterization Method Based on the TTP Criterion

Accurate grading of remote sensing image interpretation is crucial for improving image classification and screening efficiency. Through extensive research, the General Image Quality Equation (GIQE) based on the National Imagery Interpretability Rating Scale (NIIRS) has been developed. However, poor reliability and low accuracy issues remain due to the failure to consider human visual characteristics. This paper introduces the Target Task Performance (TTP) criterion as a key parameter to reflect the cascading degradation factors of human visual perception characteristics and system imaging links, which improves the reliability of the model. A New Optimized Remote Sensing Image Quality Equation (NORSIQE), which effectively predicted the interpretability of image information, is constructed. Using 200 sets of test data, the quantitative relationship between key parameters (GSD, TTP, and SNR) of the NORSIQE and the subjective NIIRS level is obtained by a least squares regression fit, and the determination coefficient of the model is as high as 0.916. The model is evaluated for accuracy using 120 sets of validation data, showing an 87% improvement compared to the GIQE4. This method provides theoretical support for the development of new methods for remote sensing image quality evaluation and the design of payloads for remote sensing imaging systems.

Towards a unified terminology for sonification and visualization.

Both sonification and visualization convey information about data by effectively using our human perceptual system, but their ways to transform the data differ. Over the past 30 years, the sonification community has demanded a holistic perspective on data representation, including audio-visual analysis, several times. A design theory of audio-visual analysis would be a relevant step in this direction. An indispensable foundation for this endeavor is a terminology describing the combined design space. To build a bridge between the domains, we adopt three of the established theoretical constructs from visualization theory for the field of sonification. The three constructs are the spatial substrate, the visual mark, and the visual channel. In our model, we choose time to be the temporal substrate of sonification. Auditory marks are then positioned in time, such as visual marks are positioned in space. Auditory channels are encoded into auditory marks to convey information. The proposed definitions allow discussing visualization and sonification designs as well as multi-modal designs based on a common terminology. While the identified terminology can support audio-visual analytics research, it also provides a new perspective on sonification theory itself.

Flying enhances viewing from above bias on ambiguous visual stimuli.

The human spatial orientation system is well designed on the ground but is imperfect in the aeronautical three-dimensional (3D) environment. However, human perception systems perform Bayesian statistics based on encountered environments and form shortcuts to improve perceptual efficiency. It is unknown whether our perception of spatial orientation is modified by flying experience and forms perceptual biases. The current study tested pilot perceptual biases on ambiguous visual stimuli, the bistable point-light walkers, and found that flying experiences increased the pilot's tendency to perceive himself as higher than the target and the target as farther away from them. Such perceptual effects due to flight are likely to be attributed to experience of variable vestibular state in a higher position in 3D space, rather than the experience of a higher viewpoint. Our findings suggest that flying experience will modifies our visual perceptual biases, and that more attention should be paid to the enhanced viewing from above bias when flying to avoid overestimating altitude or viewing angle when the visual conditions are ambiguous.