Multiple Sensory Stimuli Research Articles

Positive Emotional Responses to Socially Assistive Robots in People With Dementia: Pilot Study.

Interventions and care that can evoke positive emotions and reduce apathy or agitation are important for people with dementia. In recent years, socially assistive robots used for better dementia care have been found to be feasible. However, the immediate responses of people with dementia when they are given multiple sensory modalities from socially assistive robots have not yet been sufficiently elucidated. This study aimed to quantitatively examine the immediate emotional responses of people with dementia to stimuli presented by socially assistive robots using facial expression analysis in order to determine whether they elicited positive emotions. This pilot study adopted a single-arm interventional design. Socially assistive robots were presented to nursing home residents in a three-step procedure: (1) the robot was placed in front of participants (visual stimulus), (2) the robot was manipulated to produce sound (visual and auditory stimuli), and (3) participants held the robot in their hands (visual, auditory, and tactile stimuli). Expression intensity values for "happy," "sad," "angry," "surprised," "scared," and "disgusted" were calculated continuously using facial expression analysis with FaceReader. Additionally, self-reported feelings were assessed using a 5-point Likert scale. In addition to the comparison between the subjective and objective emotional assessments, expression intensity values were compared across the aforementioned 3 stimuli patterns within each session. Finally, the expression intensity value for "happy" was compared between the different types of robots. A total of 29 participants (mean age 88.7, SD 6.2 years; n=27 female; Japanese version of Mini-Mental State Examination mean score 18.2, SD 5.1) were recruited. The expression intensity value for "happy" was the largest in both the subjective and objective assessments and increased significantly when all sensory modalities (visual, auditory, and tactile) were presented (median expression intensity 0.21, IQR 0.09-0.35) compared to the other 2 patterns (visual alone: median expression intensity 0.10, IQR 0.03-0.22; P<.001; visual and auditory: median expression intensity 0.10, IQR 0.04-0.23; P<.001). The comparison of different types of robots revealed a significant increase when all stimuli were presented by doll-type and animal-type robots, but not humanoid-type robots. By quantifying the emotional responses of people with dementia, this study highlighted that socially assistive robots may be more effective in eliciting positive emotions when multiple sensory stimuli, including tactile stimuli, are involved. More studies, including randomized controlled trials, are required to further explore the effectiveness of using socially assistive robots in dementia care.

How to Use Classical Music to Design High-quality Infant Massage Activities

The art of music not only has performance functions, but also has application functions, especially when combined with non-music majors and disciplines. In the field of infant care services, infant massage activities are an indispensable and important part of infant care. Many caregivers in childcare institutions or families like playing music during the massage process. However, unfortunately, most people only consider music as a background and have never thought of combining the melody, rhythm, or other musical elements of music with massage techniques to design high-quality infant massage activities, allowing infants to experience multiple sensory stimuli simultaneously, further effectively and high-quality promoting the comprehensive physical and mental development of infants. The author of this article applies professional knowledge and skills in the fields of music and infant care services, combined with practical experience, and presents detailed practical cases to conduct in-depth analysis and exploration on how to use classical music to design high-quality infant massage activities.

Reflections on Cross-Modal Correspondences: Current Understanding and Issues for Future Research.

The past two decades have seen an explosion of research on cross-modal correspondences. Broadly speaking, this term has been used to encompass associations between and among features, dimensions, or attributes across the senses. There has been an increasing interest in this topic amongst researchers from multiple fields (psychology, neuroscience, music, art, environmental design, etc.) and, importantly, an increasing breadth of the topic's scope. Here, this narrative review aims to reflect on what cross-modal correspondences are, where they come from, and what underlies them. We suggest that cross-modal correspondences are usefully conceived as relative associations between different actual or imagined sensory stimuli, many of these correspondences being shared by most people. A taxonomy of correspondences with four major kinds of associations (physiological, semantic, statistical, and affective) characterizes cross-modal correspondences. Sensory dimensions (quantity/quality) and sensory features (lower perceptual/higher cognitive) correspond in cross-modal correspondences. Cross-modal correspondences may be understood (or measured) from two complementary perspectives: the phenomenal view (perceptual experiences of subjective matching) and the behavioural response view (observable patterns of behavioural response to multiple sensory stimuli). Importantly, we reflect on remaining questions and standing issues that need to be addressed in order to develop an explanatory framework for cross-modal correspondences. Future research needs (a) to understand better when (and why) phenomenal and behavioural measures are coincidental and when they are not, and, ideally, (b) to determine whether different kinds of cross-modal correspondence (quantity/quality, lower perceptual/higher cognitive) rely on the same or different mechanisms.

Statistical learning of L2 lexical bundles through unimodal, bimodal, and multimodal stimuli

Given the fundamental contribution of formulaic sequences to the fluent, natural second/foreign language (L2) use and processing, finding effective ways of improving learners’ formulaic language has increasingly engaged the attention of L2 practitioners. Focusing on lexical bundles (LBs), as the most frequent formulaic-language strings, the current quasi-experimental study examined statistical learning of recurring 3-to-6-word LBs among Iranian learners of English as a foreign language (EFL) receiving different types of distributional, narrative sensory stimuli. To this end, three intact classes received a recurring-theme narrative content in three different input modalities including only-auditory, audiovisual, and captioned audiovisual. To examine whether exposure to statistical, varied-modality input (or sensory stimuli) constituted profound knowledge of L2 lexical bundles, the three experimental groups’ performances on immediate and delayed posttests were compared with those of a control group which received no purposefully distributional, modality-based stimuli. ANOVA results revealed that all the frequency-based modality inputs led to more sizeable LB gains. Further, multiple (captioned) audiovisual sensory stimuli more effectively fostered retention and recalling of the target LBs over time than only-auditory input. The retention differences between the bimodal and multimodal inputs were insignificant. Overall, the findings supported combined statistical, formulaic advantages for processing, retention, and recalling of L2 LBs through distributional, multimodal stimuli.

Robust Moth-Inspired Algorithm for Odor Source Localization Using Multimodal Information.

Odor-source localization, by which one finds the source of an odor by detecting the odor itself, is an important ability to possess in order to search for leaking gases, explosives, and disaster survivors. Although many animals possess this ability, research on implementing olfaction in robotics is still developing. We developed a novel algorithm that enables a robot to localize an odor source indoors and outdoors by taking inspiration from the adult male silk moth, which we used as the target organism. We measured the female-localization behavior of the silk moth by using a virtual reality (VR) system to obtain the relationship between multiple sensory stimuli and behavior during the localization behavior. The results showed that there were two types of search active and inactive depending on the direction of odor and wind detection. In an active search, the silk moth moved faster as the odor-detection frequency increased, whereas in the inactive search, they always moved slower under all odor-detection frequencies. This phenomenon was constructed as a robust moth-inspired (RMI) algorithm and implemented on a ground-running robot. Experiments on odor-source localization in three environments with different degrees of environmental complexity showed that the RMI algorithm has the best localization performance among conventional moth-inspired algorithms. Analysis of the trajectories showed that the robot could move smoothly through the odor plume even when the environment became more complex. This indicates that switching and modulating behavior based on the direction of odor and wind detection contributes to the adaptability and robustness of odor-source localization.

Multi-sensory feedback improves spatially compatible sensori-motor responses

To interact with machines, from computers to cars, we need to monitor multiple sensory stimuli, and respond to them with specific motor actions. It has been shown that our ability to react to a sensory stimulus is dependent on both the stimulus modality, as well as the spatial compatibility of the stimulus and the required response. However, the compatibility effects have been examined for sensory modalities individually, and rarely for scenarios requiring individuals to choose from multiple actions. Here, we compared response time of participants when they had to choose one of several spatially distinct, but compatible, responses to visual, tactile or simultaneous visual and tactile stimuli. We observed that the presence of both tactile and visual stimuli consistently improved the response time relative to when either stimulus was presented alone. While we did not observe a difference in response times of visual and tactile stimuli, the spatial stimulus localization was observed to be faster for visual stimuli compared to tactile stimuli.

Multisensory integration and white matter pathology: Contributions to cognitive dysfunction.

The ability to simultaneously process and integrate multiple sensory stimuli is paramount to effective daily function and essential for normal cognition. Multisensory management depends critically on the interplay between bottom-up and top-down processing of sensory information, with white matter (WM) tracts acting as the conduit between cortical and subcortical gray matter (GM) regions. White matter tracts and GM structures operate in concert to manage both multisensory signals and cognition. Altered sensory processing leads to difficulties in reweighting and modulating multisensory input during various routine environmental challenges, and thus contributes to cognitive dysfunction. To examine the specific role of WM in altered sensory processing and cognitive dysfunction, this review focuses on two neurologic disorders with diffuse WM pathology, multiple sclerosis and mild traumatic brain injury, in which persistently altered sensory processing and cognitive impairment are common. In these disorders, cognitive dysfunction in association with altered sensory processing may develop initially from slowed signaling in WM tracts and, in some cases, GM pathology secondary to WM disruption, but also because of interference with cognitive function by the added burden of managing concurrent multimodal primary sensory signals. These insights promise to inform research in the neuroimaging, clinical assessment, and treatment of WM disorders, and the investigation of WM-behavior relationships.

Sex-specific brain microstructural reorganization in irritable bowel syndrome.

Preliminary evidence suggests that there are sex differences in microstructural brain organization among individuals with irritable bowel syndrome (IBS). The aim of this study was to further investigate sex-dependent differences in brain microstructure and organization in a large sample of well-phenotyped participants with IBS compared with healthy controls. We hypothesized that female patients with IBS would show evidence for increased axonal strength and myelination within and between brain regions concerned with pain and sensory processing, when compared with males with IBS. We also hypothesized that female compared with male IBS subjects show greater levels of somatic awareness and sensory sensitivity consistent with multisystem sensory sensitivity. Diffusion tensor images and clinical assessments were obtained in 100 healthy controls (61 females) and 152 IBS (107 females) on a 3T Siemens Trio. Whole brain voxel-wise differences in fractional anisotropy, mean, radial and axial diffusivity, and track density as differences in somatic awareness and sensory sensitivity were assessed using the general linear model. Female compared with male IBS participants showed extensive microstructural alterations in sensorimotor, corticothalamic, and basal ganglia circuits involved in pain processing and integration of sensorimotor information. Together with the observed increases in symptom severity, somatic awareness, and sensory sensitivity, the findings support the hypotheses that the etiology and maintenance of symptoms in females with IBS may be driven by greater central sensitivity for multiple sensory stimuli.

The Stevens law and the derivation of sensory perception

The Stevens law, S = k I n , is commonly accepted as a landmark achievement in the field of sensory psychophysics and has been widely applied in food sensory research. The theory effectively links the two parameters of very different nature, the intensity of sensory perception ( S ) and the intensity of physical stimulus ( I ). However, the Stevens law suffers from two major drawbacks. The equation was self-inconsistent in terms of the units, which makes equation questionable in principle. The Stevens law was also found inapplicable when multiple physical stimuli are involved in the perception of a sensory attribute. To solve the first drawback, we propose to replace the absolute stimulation intensity in the Stevens law with a relative intensity, S = k( I / I 0 ) n . The modified Stevens law becomes self-consistent when both sides of the equation are dimensionless and the theory is applicable regardless of the unit of the physical stimulus. To solve the second drawback, a concept of sensory derivation is introduced and so sensory properties can be categorized as primary sensory and derived sensory. The former can be described effectively by the Stevens law, while description of the latter is beyond the applicability of the Stevens law because of the involvement of multiple sensory stimuli or multi level sensory derivation. Furthermore, a sensory hierarch is constructed with sensory perception of different nature at different levels, including the primary sensory at the basic level and derived sensory perception at progressing levels. The modified Stevens law provides a self-consistent model for psychophysical analysis of sensory perception. The proposed sensory hierarchy helps to streamline sensory terms and attributes and may be used as a useful guide in establishing instrumental analysis of sensory perception.

Effects of Increasing Stimulated Area in Spatiotemporally Congruent Unisensory and Multisensory Conditions.

Research has shown that the ability to integrate complementary sensory inputs into a unique and coherent percept based on spatiotemporal coincidence can improve perceptual precision, namely multisensory integration. Despite the extensive research on multisensory integration, very little is known about the principal mechanisms responsible for the spatial interaction of multiple sensory stimuli. Furthermore, it is not clear whether the size of spatialized stimulation can affect unisensory and multisensory perception. The present study aims to unravel whether the stimulated area’s increase has a detrimental or beneficial effect on sensory threshold. Sixteen typical adults were asked to discriminate unimodal (visual, auditory, tactile), bimodal (audio-visual, audio-tactile, visuo-tactile) and trimodal (audio-visual-tactile) stimulation produced by one, two, three or four devices positioned on the forearm. Results related to unisensory conditions indicate that the increase of the stimulated area has a detrimental effect on auditory and tactile accuracy and visual reaction times, suggesting that the size of stimulated areas affects these perceptual stimulations. Concerning multisensory stimulation, our findings indicate that integrating auditory and tactile information improves sensory precision only when the stimulation area is augmented to four devices, suggesting that multisensory interaction is occurring for expanded spatial areas.

A Functional MRI Paradigm for Efficient Mapping of Memory Encoding Across Sensory Conditions.

We introduce a new and time-efficient memory-encoding paradigm for functional magnetic resonance imaging (fMRI). This paradigm is optimized for mapping multiple contrasts using a mixed design, using auditory (environmental/vocal) and visual (scene/face) stimuli. We demonstrate that the paradigm evokes robust neuronal activity in typical sensory and memory networks. We were able to detect auditory and visual sensory-specific encoding activities in auditory and visual cortices. Also, we detected stimulus-selective activation in environmental-, voice-, scene-, and face-selective brain regions (parahippocampal place and fusiform face area). A subsequent recognition task allowed the detection of sensory-specific encoding success activity (ESA) in both auditory and visual cortices, as well as sensory-unspecific positive ESA in the hippocampus. Further, sensory-unspecific negative ESA was observed in the precuneus. Among others, the parallel mixed design enabled sustained and transient activity comparison in contrast to rest blocks. Sustained and transient activations showed great overlap in most sensory brain regions, whereas several regions, typically associated with the default-mode network, showed transient rather than sustained deactivation. We also show that the use of a parallel mixed model had relatively little influence on positive or negative ESA. Together, these results demonstrate a feasible, versatile, and brief memory-encoding task, which includes multiple sensory stimuli to guarantee a comprehensive measurement. This task is especially suitable for large-scale clinical or population studies, which aim to test task-evoked sensory-specific and sensory-unspecific memory-encoding performance as well as broad sensory activity across the life span within a very limited time frame.

Metabolomics Approach to Assess the Relative Contributions of the Volatile and Non-volatile Composition to Expert Quality Ratings of Pinot Noir Wine Quality.

Wine flavor and quality are determined by the assessment of multiple sensory stimuli, including aroma, taste, and mouthfeel. It is therefore important to consider the contribution of as many metabolites as possible when attempting to relate wine composition to quality. In this study, partial least squares regression of the volatile (untargeted headspace solid-phase microextraction coupled with gas chromatography time-of-flight mass spectrometry), non-volatile (untargeted reverse-phase ultra-high-performance liquid chromatography mass spectrometry), and combined metabolite profiles were used to predict Pinot Noir wine quality ratings as assessed by experts. Non-volatile metabolite profiles predicted wine quality ratings better than volatile metabolite profiles, suggesting that the non-volatile composition of Pinot Noir wines contributes to quality perception to a greater extent than the volatile composition. This was underscored by descriptive sensory analysis, which found that taste and mouthfeel attributes were better correlated with wine quality ratings than aroma attributes. Important predictors of Pinot Noir wine quality were also characterized. Some new relationships between wine metabolites and quality ratings were found: dipeptides and unsaturated fatty acids were positively related to Pinot Noir wine quality, while N-(3-methylbutyl)acetamide and xanthine were negatively associated.

The Effects of Human Visual Sensory Stimuli on N1b Amplitude: An EEG Study.

Sensory systems are widely known to exhibit adaptive mechanisms. Vision is no exception to input dependent changes in its sensitivity. Recent animal work demonstrates enhanced connectivity between neurons in the visual cortex. The purpose of the present experiment was to evaluate a human model that noninvasively alters the amplitude of the N1b component in the visual cortex of humans by means of rapid visual stimulation. Nineteen participants (Mage = 24 years; 52.6% male) completed a rapid visual stimulation paradigm involving black and white reversal checkerboards presented bilaterally in the visual field. EEG data was collected during the visual stimulation paradigm, which consisted of four main phases, a pre-tetanus block, photic stimulus, early post-tetanus, and late post-tetanus. The amplitude of the N1b component of the pre-tetanus, early post-tetanus and late post-tetanus visual evoked potentials were calculated. Change in N1b amplitude was calculated by subtracting pre-tetanus N1b amplitude from early and late post-tetanus. Results demonstrated a significant difference between pre-tetanus N1b (M = −0.498 µV, SD = 0.858) and early N1b (M = −1.011 µV, SD = 1.088), t (18) = 2.761, p = 0.039, d = 0.633. No difference was observed between pre-tetanus N1b and late N1b (p = 0.36). In conclusion, our findings suggest that it is possible to induce changes in the amplitude of the visually evoked potential N1b waveform in the visual cortex of humans non-invasively. Additional work is needed to corroborate that the potentiation of the N1b component observed in this study is due to similar mechanisms essential to prolonged strengthened neural connections exhibited in cognitive structures of the brain observed in prior animal research. If so, this will allow for the examination of strengthened neural connectivity and its interaction with multiple human sensory stimuli and behaviors.

Stimulus salience determines defensive behaviors elicited by aversively conditioned serial compound auditory stimuli.

Assessing the imminence of threatening events using environmental cues enables proactive engagement of appropriate avoidance responses. The neural processes employed to anticipate event occurrence depend upon which cue properties are used to formulate predictions. In serial compound stimulus (SCS) conditioning in mice, repeated presentations of sequential tone (CS1) and white noise (CS2) auditory stimuli immediately prior to an aversive event (US) produces freezing and flight responses to CS1 and CS2, respectively (Fadok et al., 2017). Recent work reported that these responses reflect learned temporal relationships of CS1 and CS2 to the US (Dong et al., 2019). However, we find that frequency and sound pressure levels, not temporal proximity to the US, are the key factors underlying SCS-driven conditioned responses. Moreover, white noise elicits greater physiological and behavioral responses than tones even prior to conditioning. Thus, stimulus salience is the primary determinant of behavior in the SCS paradigm, and represents a potential confound in experiments utilizing multiple sensory stimuli.

4D functional ultrasound imaging of whole-brain activity in rodents.

We developed a technology that extends the capabilities of functional ultrasound to whole-brain 4D neuroimaging. We implemented multiplane wave transmissions on matrix arrays at thousands of frames per second to provide volumetric recordings of cerebral blood volume changes at high spatiotemporal resolution. This approach is illustrated by several proofs-of-concept: real time multiple sensory stimuli, 4D functional connectivity and instantaneous tracking of epileptiform events within the whole brains of rodents.

First Error-Based Supervised Learning Algorithm for Spiking Neural Networks.

Neural circuits respond to multiple sensory stimuli by firing precisely timed spikes. Inspired by this phenomenon, the spike timing-based spiking neural networks (SNNs) are proposed to process and memorize the spatiotemporal spike patterns. However, the response speed and accuracy of the existing learning algorithms of SNNs are still lacking compared to the human brain. To further improve the performance of learning precisely timed spikes, we propose a new weight updating mechanism which always adjusts the synaptic weights at the first wrong output spike time. The proposed learning algorithm can accurately adjust the synaptic weights that contribute to the membrane potential of desired and non-desired firing time. Experimental results demonstrate that the proposed algorithm shows higher accuracy, better robustness, and less computational resources compared with the remote supervised method (ReSuMe) and the spike pattern association neuron (SPAN), which are classic sequence learning algorithms. In addition, the SNN-based computational model equipped with the proposed learning method achieves better recognition results in speech recognition task compared with other bio-inspired baseline systems.

Toward Lifelong Affordance Learning Using a Distributed Markov Model

Robots are able to learn how to interact with objects by developing computational models of affordance. This paper presents an approach in which learning and operation occur concurrently, toward achieving lifelong affordance learning. In a such a regime a robot must be able to learn about new objects, but without a general rule for what an “object” is, the robot must learn about everything in the environment to determine their affordances. In this paper, sensorimotor coordination is modeled using a distributed semi-Markov decision process; it is created online during robot operation, and performs continual action selection to reach a goal state. In an initial experiment we show that this model captures an object’s affordances, which are exploited to perform several different tasks using a mobile robot equipped with a gripper and infrared “tactile” sensor. In a secondary experiment, we show that the robot can learn that the marker is the only visual feature that can be gripped and that walls and floor do not have the affordance of being “grip-able.” The distributed mechanism is necessary for the modeling of multiple sensory stimuli simultaneously, and the selection of the object with the necessary affordances for the task is emergent from the robot’s actions, while other parts of the environment that are perceived, such as walls and floors, are ignored.

A neural mechanism of cognitive control for resolving conflict between abstract task rules

Conflict between multiple sensory stimuli or potential motor responses is thought to be resolved via bias signals from prefrontal cortex (PFC). However, population codes in the PFC also represent abstract information, such as task rules. How is conflict between active abstract representations resolved? We used functional neuroimaging to investigate the mechanism responsible for resolving conflict between abstract representations of task rules. Participants performed two different tasks based on a cue. We manipulated the degree of conflict at the task-rule level by training participants to associate the color and shape dimensions of the cue with either the same task rule (congruent cues) or different ones (incongruent cues). Phonological and semantic tasks were used in which performance depended on learned, abstract representations of information, rather than sensory features of the target stimulus or on any habituated stimulus-response associations. In addition, these tasks activate distinct regions that allowed us to measure magnitude of conflict between tasks. We found that incongruent cues were associated with increased activity in several cognitive control areas, including the inferior frontal gyrus, inferior parietal lobule, insula, and subcortical regions. Conflict between abstract representations appears to be resolved by rule-specific activity in the inferior frontal gyrus that is correlated with enhanced activity related to the relevant information. Furthermore, multi-voxel pattern analysis of the activity in the inferior frontal gyrus was shown to carry information about both the currently relevant rule (semantic/phonological) and the currently relevant cue context (color/shape). Similar to models of attentional selection of conflicting sensory or motor representations, the current findings indicate part of the frontal cortex provides a bias signal, representing task rules, that enhances task-relevant information. However, the frontal cortex can also be the target of these bias signals in order to enhance abstract representations that are independent of particular stimuli or motor responses.

Hunger-Driven Motivational State Competition

Behavioral choice is ubiquitous in the animal kingdom and is central to goal-oriented behavior. Hypothalamic Agouti-related peptide (AgRP) neurons are critical regulators of appetite. Hungry animals, bombarded by multiple sensory stimuli, are known to modify their behavior during times of caloric need, rapidly adapting to a consistently changing environment. Utilizing ARCAgRP neurons as an entry point, we analyzed the hierarchical position of hunger related to rival drive states. Employing a battery ofbehavioral assays, we found that hunger significantly increases its capacity to suppress competing motivational systems, such as thirst, anxiety-related behavior, innate fear, and social interactions, often only when food is accessible. Furthermore, real-time monitoring of ARCAgRP activity revealed time-locked responses to conspecific investigation in addition to food presentation, further establishing that, even at the level of ARCAgRP neurons, choices are remarkablyflexible computations, integrating internal state, external factors, and anticipated yield. VIDEO ABSTRACT.

A review of human sensory dynamics for application to models of driver steering and speed control.

In comparison with the high level of knowledge about vehicle dynamics which exists nowadays, the role of the driver in the driver–vehicle system is still relatively poorly understood. A large variety of driver models exist for various applications; however, few of them take account of the driver’s sensory dynamics, and those that do are limited in their scope and accuracy. A review of the literature has been carried out to consolidate information from previous studies which may be useful when incorporating human sensory systems into the design of a driver model. This includes information on sensory dynamics, delays, thresholds and integration of multiple sensory stimuli. This review should provide a basis for further study into sensory perception during driving.