Human Sensory System Research Articles

Human-Like Modulation Sensitivity Emerging through Optimization to Natural Sound Recognition.

Natural sounds contain rich patterns of amplitude modulation (AM), which is one of the essential sound dimensions for auditory perception. The sensitivity of human hearing to AM measured by psychophysics takes diverse forms depending on the experimental conditions. Here, we address with a single framework the questions of why such patterns of AM sensitivity have emerged in the human auditory system and how they are realized by our neural mechanisms. Assuming that optimization for natural sound recognition has taken place during human evolution and development, we examined its effect on the formation of AM sensitivity by optimizing a computational model, specifically, a multilayer neural network, for natural sound (namely, everyday sounds and speech sounds) recognition and simulating psychophysical experiments in which the AM sensitivity of the model was assessed. Relatively higher layers in the model optimized to sounds with natural AM statistics exhibited AM sensitivity similar to that of humans, although the model was not designed to reproduce human-like AM sensitivity. Moreover, simulated neurophysiological experiments on the model revealed a correspondence between the model layers and the auditory brain regions. The layers in which human-like psychophysical AM sensitivity emerged exhibited substantial neurophysiological similarity with the auditory midbrain and higher regions. These results suggest that human behavioral AM sensitivity has emerged as a result of optimization for natural sound recognition in the course of our evolution and/or development and that it is based on a stimulus representation encoded in the neural firing rates in the auditory midbrain and higher regions.SIGNIFICANCE STATEMENT This study provides a computational paradigm to bridge the gap between the behavioral properties of human sensory systems as measured in psychophysics and neural representations as measured in nonhuman neurophysiology. This was accomplished by combining the knowledge and techniques in psychophysics, neurophysiology, and machine learning. As a specific target modality, we focused on the auditory sensitivity to sound AM. We built an artificial neural network model that performs natural sound recognition and simulated psychophysical and neurophysiological experiments in the model. Quantitative comparison of a machine learning model with human and nonhuman data made it possible to integrate the knowledge of behavioral AM sensitivity and neural AM tunings from the perspective of optimization to natural sound recognition.

Carbon Nanodots Memristor: An Emerging Candidate toward Artificial Biosynapse and Human Sensory Perception System.

In the era of big data and artificial intelligence (AI), advanced data storage and processing technologies are in urgent demand. The innovative neuromorphic algorithm and hardware based on memristor devices hold a promise to break the von Neumann bottleneck. In recent years, carbon nanodots (CDs) have emerged as a new class of nano-carbon materials, which have attracted widespread attention in the applications of chemical sensors, bioimaging, and memristors. The focus of this review is to summarize the main advances of CDs-based memristors, and their state-of-the-art applications in artificial synapses, neuromorphic computing, and human sensory perception systems. The first step is to systematically introduce the synthetic methods of CDs and their derivatives, providing instructive guidance to prepare high-quality CDs with desired properties. Then, the structure-property relationship and resistive switching mechanism of CDs-based memristors are discussed in depth. The current challenges and prospects of memristor-based artificial synapses and neuromorphic computing are also presented. Moreover, this review outlines some promising application scenarios of CDs-based memristors, including neuromorphic sensors and vision, low-energy quantum computation, and human-machine collaboration.

Nonverbal Cues in Human–Robot Interaction: A Communication Studies Perspective

Communication between people is characterized by a broad range of nonverbal cues. Transferring these cues into the design of robots and other artificial agents that interact with people may foster more natural, inviting, and accessible experiences. In this article, we offer a series of definitive nonverbal codes for human–robot interaction (HRI) that address the five human sensory systems (visual, auditory, haptic, olfactory, and gustatory) drawn from the field of communication studies. We discuss how these codes can be translated into design patterns for HRI using a curated sample of the communication studies and HRI literatures. As nonverbal codes are an essential mode in human communication, we argue that integrating robotic nonverbal codes in HRI will afford robots a feeling of “aliveness” or “social agency” that would otherwise be missing. We end with suggestions for research directions to stimulate work on nonverbal communication within the field of HRI and improve communication between people and robots.

A Modular In-Vehicle C-ITS Architecture for Sensor Data Collection, Vehicular Communications and Cloud Connectivity.

The growth of the automobile industry in recent decades and the overuse of personal vehicles have amplified problems directly related to road safety, such as the increase in traffic congestion and number of accidents, as well as the degradation of the quality of roads. At the same time, and with the contribution of climate change effects, dangerous weather events have become more common on road infrastructure. In this context, Cooperative Intelligent Transport Systems (C-ITS) and Internet of Things (IoT) solutions emerge to overcome the limitations of human and local sensory systems, through the collection and distribution of relevant data to Connected and Automated Vehicles (CAVs). In this paper, an intra- and inter-vehicle sensory data collection system is presented, starting with the acquisition of relevant data present on the Controller Area Network (CAN) bus, collected through the vehicle's On-Board-Diagnostics II (OBD-II) port, as well as on an on-board smartphone device and possibly other additional sensors. Short-range communication technologies, such as Bluetooth Low Energy (BLE), Wi-Fi, and ITS-G5, are employed in conjunction with long-range cellular networks for data dissemination and remote cloud monitoring. The results of the experimental tests allow the analysis of the road environment, as well as the notification in near real-time of adverse road conditions to drivers. The developed data collection system reveals itself as a potentially valuable tool for improving road safety and to iterate on the current Road Weather Models (RWMs).

A new path of visual communication from the perspective of the metaverse —— Take Liu Yexi’s Internet Communication Spectacle as an Example

In 2021, the “Metaverse” concept became a big hit. As an exceptional media concept, the metaverse is closely related to the human sensory system, especially visual communication. Through the method of literature research and case analysis, combined with the recent publicity craze of “Liu Yexi” and from the perspective of the combination of the metaverse and short video, this paper argues that the richness and abstraction of the concept of metaverse require short video as a new form of communication. On this basis, the practical experience of “Liu Yexi” visual communication is analyzed from narrative logic, innovation leap, and visual spectacle. Then it is pointed out that the capitalization trend of artificial intelligence is the potential risk the metaverse may face in the long-term development.

Efficient neural codes naturally emerge through gradient descent learning

Human sensory systems are more sensitive to common features in the environment than uncommon features. For example, small deviations from the more frequently encountered horizontal orientations can be more easily detected than small deviations from the less frequent diagonal ones. Here we find that artificial neural networks trained to recognize objects also have patterns of sensitivity that match the statistics of features in images. To interpret these findings, we show mathematically that learning with gradient descent in neural networks preferentially creates representations that are more sensitive to common features, a hallmark of efficient coding. This effect occurs in systems with otherwise unconstrained coding resources, and additionally when learning towards both supervised and unsupervised objectives. This result demonstrates that efficient codes can naturally emerge from gradient-like learning.

Robot Hearing Through Optical Channel in a Cocktail Party Environment

The cocktail party problem refers to a challenging process when the human sensory system tries to separate a specific voice from a loud mixture of background sound sources. The problem is much more demanding for machines and has become the holy grail in robotic hearing. Despite many advances in noise suppression, the intrinsic information from the contaminated acoustic channel remains difficult to recover. Herein, a simple‐yet‐powerful laser‐assisted audio system termed robot ear accomplished by laser (REAL) is shown to probe the vibrations of sound‐carrying surfaces (mask, throat, and other nearby surfaces) in optical channel, which is intrinsically immune to acoustic background noises. The results demonstrate that REAL can directly obtain the audio‐frequency content from the laser without acoustic channel interference. The signals can be further transcribed into human‐recognizable audios by exploiting the internal time and frequency correlations through memory‐enabled neural networks. The REAL system would enable a new way in human–robot interaction. An interactive preprint version of the article can be found at: https://www.authorea.com/doi/full/10.1002/aisy.202200143.

Pictures from the Atomic Force Microscope: Temporal Representation in the Moving Image

Historically, artists and scientists predominantly created moving images of scientific phenomena using optical instrumentation such as cameras and light microscopes that enable direct observation of their subject. However, the perception and representation of nanoscale phenomena, which are too small to be detected by optical systems, rely upon complex technological mediation via layers of instrumentation, hardware and software. This mediation has profound implications for the perception of phenomena that exist outside the range of the human sensory system. This article describes the scientific representations of temporality in the work of Eadweard Muybridge and Étienne-Jules Marey and the complications of visualising the nanoscale. I analyse imperceptible phenomena in the contemporary moving image and draw upon my creative practice with the Atomic Force Microscope (AFM), using this instrument as a framework for discussing the temporal and sensorial challenges of working with nanoscale phenomena. This creative practice-based research resulted in the creation of several experimental moving image works collected under the title Wildly Oscillating Molecules. Through the adoption of scientific instrumentation and data for creative production, the collection of moving image practices discussed uncover temporal and sensorial assumptions and suggest alternative experiential encounters with scientific phenomena.

The spatial profile of skin indentation shapes tactile perception across stimulus frequencies

Multiple human sensory systems exhibit sensitivity to spatial and temporal variations of physical stimuli. Vision has evolved to offer high spatial acuity with limited temporal sensitivity, while audition has developed complementary characteristics. Neural coding in touch has been believed to transition from a spatial to a temporal domain in relation to surface scale, such that coarse features (e.g., a braille cell or corduroy texture) are coded as spatially distributed signals, while fine textures (e.g., fine-grit sandpaper) are encoded by temporal variation. However, the interplay between the two domains is not well understood. We studied tactile encoding with a custom-designed pin array apparatus capable of deforming the fingerpad at 5 to 80 Hz in each of 14 individual locations spaced 2.5 mm apart. Spatial variation of skin indentation was controlled by moving each of the pins at the same frequency and amplitude, but with phase delays distributed across the array. Results indicate that such stimuli enable rendering of shape features at actuation frequencies up to 20 Hz. Even at frequencies > 20 Hz, however, spatial variation of skin indentation continues to play a vital role. In particular, perceived roughness is affected by spatial variation within the fingerpad even at 80 Hz. We provide evidence that perceived roughness is encoded via a summary measure of skin displacement. Relative displacements in neighboring pins of less than 10 µm generate skin stretch, which regulates the roughness percept.

Soft, All‐Polymer Optoelectronic Tactile Sensor for Stick‐Slip Detection

AbstractThe mechanoreceptors of the human tactile sensory system contribute to natural grasping manipulations in everyday life. However, in the case of robot systems, attempts to emulate humans’ dexterity are still limited by tactile sensory feedback. In this work, a soft optical lightguide is applied as an afferent nerve fiber in a tactile sensory system. A skin‐like soft silicone material is combined with a bristle friction model, which is capable of fast and easy fabrication. Due to this novel design, the soft sensor can provide not only normal force (up to 5 Newtons) but also lateral force information generated by stick‐slip processes. Through a static force test and slip motion test, its ability to measure normal forces and to detect stick‐slip events is demonstrated. Finally, using a robotic gripper, real‐time control applications are investigated where the sensor helps the gripper apply sufficient force to grasp objects without slipping.

Breaking the Limits of Display and Fabrication using Perception-aware Optimizations

Novel display devices and fabrication techniques enable highly tangible ways of creating, experiencing, and interacting with digital content. The capabilities offered by these new output devices, such as virtual and augmented reality head-mounted displays and new multi-material 3D printers, make them real game-changers in many fields. At the same time, the new possibilities offered by these devices impose many challenges for content creation techniques regarding quality and computational efficiency. This paper discusses the concept of perception-aware optimizations, which incorporate insights from human perception into computational methods to optimize content according to the capabilities of different output devices, e.g., displays, 3D printers, and requirements of the human sensory system. As demonstrated in this paper, the key advantage of such strategies is that tailoring computation to perceptually-relevant aspects of the content often reduces the computational cost related to the content creation or overcomes certain limitations of output devices. Besides discussing the general concept, the paper presents several specific applications where perception-aware optimization has been proven beneficial. The examples include methods for optimizing visual content for novel display devices that focus on perceived quality and new computational fabrication techniques for manufacturing objects that look and feel like real ones.

Hybrid electronic skin combining triboelectric nanogenerator and humidity sensor for contact and non-contact sensing

Developing an electronic skin (e-skin) by mimicking a human skin sensory system in which tactile sensing and non-contact sensing complement each is of great significance for helping wearable electronic products to realize the intelligent perception of contact and non-contact information. Here, a hybrid e-skin (CNES) that realizes both contact and non-contact sensing is reported, which consists of a triboelectric nanogenerator (TENG) and a humidity sensor that share a flexible substrate layer. The shared flexible substrate is formed by a hydrothermal method and presents a porous and nanoflake-like morphology surface. By introducing an Ag nanowires (NWs) electrification layer on the upper surface of the shared substrate, CNES exhibits advantages in contact sensing and electrical output performance. Therefore, we achieve the monitoring for falling behaviors with such a CNES serving as a contact sensor and the management of humidity management system with the matrixes of CNESs simulating the self-powered switches. In addition, the SnO2 humidity sensing layer that is introduced on the lower surface of the shared substrate facilitates the CNES to achieve the non-contact monitoring of human respiration signals, offering a reliable pathway to design electronic skin for both contact and non-contact sensing.

Automation of reversible steganographic coding with nonlinear discrete optimisation

Authentication mechanisms are at the forefront of defending the world from various types of cybercrime. Steganography can serve as an authentication solution through the use of a digital signature embedded in a carrier object to ensure the integrity of the object and simultaneously lighten the burden of metadata management. Nevertheless, despite being generally imperceptible to human sensory systems, any degree of steganographic distortion might be inadmissible in fidelity-sensitive situations such as forensic science, legal proceedings, medical diagnosis and military reconnaissance. This has led to the development of reversible steganography. A fundamental element of reversible steganography is predictive analytics, for which powerful neural network models have been effectively deployed. Another core element is reversible steganographic coding. Contemporary coding is based primarily on heuristics, which offers a shortcut towards sufficient, but not necessarily optimal, capacity–distortion performance. While attempts have been made to realise automatic coding with neural networks, perfect reversibility is unattainable via such learning machinery. Instead of relying on heuristics and machine learning, we aim to derive optimal coding by means of mathematical optimisation. In this study, we formulate reversible steganographic coding as a nonlinear discrete optimisation problem with a logarithmic capacity constraint and a quadratic distortion objective. Linearisation techniques are developed to enable iterative mixed-integer linear programming. Experimental results validate the near-optimality of the proposed optimisation algorithm when benchmarked against a brute-force method.

Development Of A Website Based On A Contextual Approach As A Biology Learning Media On The Subconcept Of The Human Sensory System

Development Of A Website Based On A Contextual Approach As A Biology Learning Media On The Subconcept Of The Human Sensory System

Sensory Experiences and Children With Severe Disabilities: Impacts on Learning.

The human sensory system is continuously engaged in experiencing and interpreting every interaction with other living beings, objects, and the environment. The purpose of this article is to describe the impact limited opportunities for rich sensory experiences have on students with severe disabilities in two middle school classrooms situated in a public separate school in the southeastern USA. The study employed a postcritical ethnographic approach and grounded theory thematic analysis of fieldnotes gathered over a two-year period. Three major themes supported by the data are presented and discussed in depth. They are: (a) students are afforded limited sensory rich experiences, (b) everyday routines make students passive recipients to school, and (c) instructional approaches result in little interaction with extended periods of waiting. The implications of the findings for improved sensory experiences and possible future directions are described.

Dynamic Measurement of Portos Tomato Seedling Growth Using the Kinect 2.0 Sensor

Traditionally farmers monitor their crops employing their senses and experience. However, the human sensory system is inconsistent due to stress, health, and age. In this paper, we propose an agronomic application for monitoring the growth of Portos tomato seedlings using Kinect 2.0 to build a more accurate, cost-effective, and portable system. The proposed methodology classifies the tomato seedlings into four categories: The first corresponds to the seedling with normal growth at the time of germination; the second corresponds to germination that occurred days after; the third category entails exceedingly late germination where its growth will be outside of the estimated harvest time; the fourth category corresponds to seedlings that did not germinate. Typically, an expert performs this classification by analyzing ten percent of the randomly selected seedlings. In this work, we studied different methods of segmentation and classification where the Gaussian Mixture Model (GMM) and Decision Tree Classifier (DTC) showed the best performance in segmenting and classifying Portos tomato seedlings.

Oxygen-Vacancy-Induced Synaptic Plasticity in an Electrospun InGdO Nanofiber Transistor for a Gas Sensory System with a Learning Function.

The perceptual learning function of a simulating human body is very important for constructing a neural computing system and a brainlike computer in the future. The sense of smell is an important part of the human sensory nervous system. However, current gas sensors simply convert gas concentrations into electrical signals and do not have the same learning and memory function as synapses. To solve this problem, we propose a new sensing idea to induce and activate the synaptic properties of transistors by adjusting the oxygen vacancy in the active layer. This sensor combines gas detection with synaptic memory and learning and overcomes the disadvantage of the separation of synaptic transistors and sensors, thus greatly reducing the cost of production. This work combines the detection of N,N-dimethylformamide (DMF) gas with the synaptic mechanism of human olfactory nerves. We successfully fabricated an InGdO nanofiber field-effect transistor by electrostatic spinning and simulated the response of human olfactory synapses to target gas by regulating the oxygen vacancy of the InGdO nanofiber. The synaptic transistor response under different concentrations of unmodulated pulses is tested, and the pavlovian conditioned reflex experiment is simulated successfully. This work provides a new idea of a gas sensor device, which is very important for the development of high-performance gas sensors and bionic electronic devices in the future.

A Visible Video Data Hiding Scheme Based on Fade-In and Fade-Out Effects Utilizing Barcodes

Data hiding techniques are usually applied into digital watermarking or digital fingerprinting, which is used to protect intellectual property rights or to avoid illegal copies of the original works. It has been pointed out that data hiding can be utilized as a communication medium. In conventional digital watermarking frameworks, it is required that the difference between the cover objects and the stego objects are quite small, such that the difference cannot be recognized by human sensory systems. On the other hand, the authors have proposed a ‘hearable’ data hiding technique for audio signals that can carry secret messages and can be naturally recognized as a musical piece by human ears. In this study, we extend the idea of the hearable data hiding into video signals by utilizing the visual effects. As visual effects, we employ fade-in and fade-out effects which can be used as a kind of visual rendering for scene transitions. In the proposed schemes, secret messages are generated as one-dimensional barcodes which are used for fade-in or fade-out effects. The present paper shows that the proposed schemes have the high accuracy in extracting the embedded messages even from the video signals captured by smartphones or tablets. It is also shown that the video signals conveying the embedded messages can be naturally recognized by human visual systems through subjective evaluation experiments.

Artificial intelligence in oncologic imaging

Radiology is integral to cancer care. Compared to molecular assays, imaging has its advantages. Imaging as a noninvasive tool can assess the entirety of tumor unbiased by sampling error and is routinely acquired at multiple time points in oncological practice. Imaging data can be digitally post-processed for quantitative assessment. The ever-increasing application of Artificial intelligence (AI) to clinical imaging is challenging radiology to become a discipline with competence in data science, which plays an important role in modern oncology. Beyond streamlining certain clinical tasks, the power of AI lies in its ability to reveal previously undetected or even imperceptible radiographic patterns that may be difficult to ascertain by the human sensory system. Here, we provide a narrative review of the emerging AI applications relevant to the oncological imaging spectrum and elaborate on emerging paradigms and opportunities. We envision that these technical advances will change radiology in the coming years, leading to the optimization of imaging acquisition and discovery of clinically relevant biomarkers for cancer diagnosis, staging, and treatment monitoring. Together, they pave the road for future clinical translation in precision oncology.

A wireless "Janus" soft gripper with multiple tactile sensors.

Biomimetic properties allow soft robots to complexly interact with the environment. As the bridge between the robot and the operating object, the gripping hand is an important organ for its connection with the outside world, which requires the ability to provide feedback from the grasped object, similar to the human sensory and nervous system. In this work, to cope with the difficulty of integrating complex sensing and communication systems into flexible soft grippers, we propose a GO/PI composite bilayer film-based gripper with two types of tactile sensors and a LC passive wireless transmission module to obtain the grip information and transmit it to the processor. The bilayer film structure demonstrates good photothermal driving performance. Pressure and material sensors are located at the tips of the gripper's fingers to acquire tactile information which is wirelessly transmitted to the processor for analysis via the LC circuit. The grasping and feedback of the gripper are presented through an intelligent display system, realizing the wireless interconnection between the robot terminal and processing system, exhibiting broad application potential.