Stimulus Modality Research Articles

Semantic language decoding across participants and stimulus modalities.

Brain decoders that reconstruct language from semantic representations have the potential to improve communication for people with impaired language production. However, training a semantic decoder for a participant currently requires many hours of brain responses to linguistic stimuli, and people with impaired language production often also have impaired language comprehension. In this study, we tested whether language can be decoded from a goal participant without using any linguistic training data from that participant. We trained semantic decoders on brain responses from separate reference participants and then used functional alignment to transfer the decoders to the goal participant. Cross-participant decoder predictions were semantically related to the stimulus words, even when functional alignment was performed using movies with no linguistic content. To assess how much semantic representations are shared between language and vision, we compared functional alignment accuracy using story and movie stimuli and found that performance was comparable in most cortical regions. Finally, we tested whether cross-participant decoders could be robust to lesions by excluding brain regions from the goal participant prior to functional alignment and found that cross-participant decoders do not depend on data from any single brain region. These results demonstrate that cross-participant decoding can reduce the amount of linguistic training data required from a goal participant and potentially enable language decoding from participants who struggle with both language production and language comprehension.

The role of left anterior temporal lobe in basic linguistic composition: Evidence from stereo-electroencephalography.

Determining the neural bases of basic linguistic composition is central to research on the cognitive neuroscience of language. The left anterior temporal lobe (LATL) is widely reported during linguistic composition of visual stimuli in magnetoencephalography (MEG) studies. However, this effect is less reported during the linguistic composition of auditory stimuli in intracranial electroencephalography (iEEG) studies. To further test this discrepancy, we directly examined the composition effect in LATL by combining the iEEG technique with visual stimuli. Participants were asked to read minimal two-word phrases in the composition condition and read words preceded by pound signs in the noncomposition condition. The results showed that high-gamma power in the LATL was higher in the composition condition compared to the noncomposition condition. These results provide more substantial evidence for the role of LATL in basic linguistic composition (at least in visual modality) and highlight the potential role of stimuli modality (visual vs. auditory) in the phrasal composition effect in LATL.

Are you tuned in to others' mind? A cross-modal evaluation of affective theory of mind in people with Parkinson's disease.

Affective Theory of Mind (ToM) is the ability to understand other peoples' emotional states and feelings. Several studies showed impaired affective ToM abilities in people with Parkinson's disease (PD). However, most studies tested this ability by using single-stimulus modality tasks (visual cues). The aim of the present study was to evaluate whether affective ToM abilities are impaired in PD using multiple stimulus modalities and whether they are related to disease duration and cognitive/emotional abilities. Twenty mid-stage, non-demented people with PD and 20 matched controls were evaluated by means of two tasks requiring subjects to infer others' mental states from the eyes' expression - the Reading the Mind in the Eyes (RME) test, and from the emotional prosody - a modified version of the Reading the Mind in the Voice (RMV) test. In people with PD, cognitive function and emotional processes were assessed through cognitive tasks and batteries of facial and prosodic emotion recognition tests. Compared to controls, the PD group showed significantly impaired performances in both RME and RMV tests (p<0.0001). Affective ToM abilities in people with PD were not correlated with cognitive function or disease duration. Visual recognition of anger and auditory recognition of fear were positively correlated with RME (p=0.0028) and RMV (p<0.0001) test scores, respectively. Cross-modal impairment of affective ToM abilities is a non-motor feature of PD, unrelated to disease stage or cognitive status. A dysfunctional amygdala-centred network might represent the shared bases for impairments in fear and anger recognition and affective ToM abilities in PD.

Multitasking Practice Eliminates Modality-Based Interference by Separating Task Representations in Sensory Brain Regions.

The debate on the neural basis of multitasking costs evolves around neural overlap between concurrently performed tasks. Recent evidence suggests that training-related reductions in representational overlap in fronto-parietal brain regions predict multitasking improvements. Cognitive theories assume that overlap of task representations may lead to unintended information exchange between tasks (i.e., crosstalk). Modality-based crosstalk was suggested as a source for multitasking costs in multisensory settings. Robust findings of increased costs for certain modality mappings may be explained by crosstalk between the stimulus modality in one task and sensory action consequences in the concurrently performed task. Whether modality-based crosstalk emerges from representational overlap in general fronto-parietal multitasking regions or modality-specific regions is not known yet. In this functional neuroimaging study, we investigate neural overlap during multitasking performance in humans, focusing on modality compatibility by employing multivariate pattern analysis and modality-specific practice interventions in three groups (total N = 54, 24 females). We observed significant differences between modality compatible and modality incompatible single-task representations, specifically in the auditory cortex but not in fronto-parietal regions. Notably, improved auditory decoding accuracy related to modality incompatible tasks was predictive of performance gains in the corresponding dual task along with complete elimination of modality-specific dual-task costs. This predictive relationship was evident only in the group practicing modality incompatible mappings, suggesting that specific practice on task sets with modality overlap influenced both neural representations and subsequent multitasking performance. This study contributes to the integration of cognitive theory and neuroscience and the role of task representations in dual-task interference.

Radiofrequency evoked potentials: A new window into the nociceptive system.

To describe the cortical evoked potentials in response to radiofrequency stimulation (RFEPs) in human volunteers. Seventeen healthy volunteers participated in an experimental session in which radiofrequency (RF) and electrical (ES) stimulation were applied to the dorsum of the hands and feet. EEG was recorded to evaluate evoked responses for each stimulus modality and stimulation site. Electrophysiological results showed highly synchronous responses compatible with activation of heat-sensitive nociceptors. Latencies of the N2 and P2 peaks in the RFEPs were longer compared to EPs evoked by ES. Furthermore, the latency of the P2 peak was also longer after stimulation of the feet compared to the hand. RF stimulation is capable of selective activation of nociceptive fibres by means of rapid skin heating. RFEPs showed the highest degree of synchronicity achieved to date for evoked cortical responses to thermal stimulation. RF stimulation represents a viable alternative in the experimental and clinical assessment of the nociceptive system.

Information Retrieval Performance on Story Recall in Normal Aging

Objectives: The present study aimed to determine whether stimulus modality, age group, and time of performance differed in story recall performance in normal aging. Methods: Seventy-three healthy older adults were divided into two groups, 65-74 years old and 75+ years old, and randomly assigned to an auditory condition and an audiovisual condition to perform a story recall task. For each independent variable, event sequencing scores and total number of information units (IUs) were compared, and information-finding behaviors were categorized into reformulation, repetition, filler, and insertion in order to compare the frequency of production by type of information-finding behavior.Results: First, there was no significant difference between groups in event sequencing score according to modalities, age, and time interval. Second, there was no significant difference between groups in the total number of IUs according to modalities, age, and time interval, but all groups performed better on average in the second trials than in the first. Third, the frequencies of all types of information-finding behaviors decreased in the second trials compared to the first, and there was a significant difference between time intervals, especially in fillers and insertions. Conclusion: This study provides meaningful information about variables that should be considered when establishing norms for story recall assessment.

Cue modality modulates interaction between exogenous spatial attention and audiovisual integration.

Exogenous spatial attention attenuates audiovisual integration (AVI). Previous studies on the effects of exogenous spatial attention on AVI have focused on the inhibition of return (IOR) effect induced by visual cues and the facilitation effect induced by auditory cues, but the differences between the effects of exogenous spatial attention (induced by visual and auditory cues) on AVI remain unclear. The present study used the exogenous spatial cue-target paradigm and manipulated cue stimulus modality (visual cue, auditory cue) in two experiments (Experiment 1: facilitation effect; Experiment 2: IOR effect) to examine the effects of exogenous spatial attention (evoked by cues in different modalities) on AVI. The results of Experiment 1 showed that the AVI effect at valid cue locations was significantly lower than that at invalid cue locations in both visual and auditory cue conditions, suggesting that the exogenous spatial facilitation effect evoked by both visual and auditory cues attenuated AVI. Further analysis showed that the facilitation effect induced by visual cues attenuated AVI to a greater extent than that induced by auditory cues. In Experiment 2, the AVI effect was significantly lower at valid cue locations than at invalid cue locations in the visual cue condition, whereas there was no significant difference in AVI effect between valid and invalid cue locations in the auditory cue condition, suggesting that the exogenous spatial IOR effect evoked by visual cues attenuated AVI, while the IOR effect evoked by auditory cues had no significant effect on AVI. Taken together, these results may suggest that exogenous spatial attention induced by visual cues has a greater effect on AVI than that induced by auditory cues.

Multisensory integration enhances audiovisual responses in the Mauthner cell.

Multisensory integration (MSI) combines information from multiple sensory modalities to create a coherent perception of the world. In contexts where sensory information is limited or equivocal, it also allows animals to integrate individually ambiguous stimuli into a clearer or more accurate percept and, thus, react with a more adaptive behavioral response. Although responses to multisensory stimuli have been described at the neuronal and behavioral levels, a causal or direct link between these two is still missing. In this study, we studied the integration of audiovisual inputs in the Mauthner cell, a command neuron necessary and sufficient to trigger a stereotypical escape response in fish. We performed intracellular recordings in adult goldfish while presenting a diverse range of stimuli to determine which stimulus properties affect their integration. Our results show that stimulus modality, intensity, temporal structure, and interstimulus delay affect input summation. Mechanistically, we found that the distinct decay dynamics of FFI triggered by auditory and visual stimuli can account for certain aspects of input integration. Altogether, this is a rare example of the characterization of MSI in a cell with clear behavioral relevance, providing both phenomenological and mechanistic insights into how MSI depends on stimulus properties.

Attentional modulation of secondary somatosensory and visual thalamus of mice.

Each sensory modality has its own primary and secondary thalamic nuclei. While the primary thalamic nuclei are well understood to relay sensory information from the periphery to the cortex, the role of secondary sensory nuclei is elusive. We trained head-fixed mice to attend to one sensory modality while ignoring a second modality, namely to attend to touch and ignore vision, or vice versa. Arrays were used to record simultaneously from the secondary somatosensory thalamus (POm) and secondary visual thalamus (LP). In mice trained to respond to tactile stimuli and ignore visual stimuli, POm was robustly activated by touch and largely unresponsive to visual stimuli. A different pattern was observed when mice were trained to respond to visual stimuli and ignore touch, with POm now more robustly activated during visual trials. This POm activity was not explained by differences in movements (i.e. whisking, licking, pupil dilation) resulting from the two tasks. Post hoc histological reconstruction of array tracks through POm revealed that subregions varied in their degree of plasticity. LP exhibited similar phenomena. We conclude that behavioral training reshapes activity in secondary thalamic nuclei. Secondary nuclei respond to the same behaviorally relevant, reward-predicting stimuli regardless of stimulus modality.

Attentional modulation of secondary somatosensory and visual thalamus of mice

Each sensory modality has its own primary and secondary thalamic nuclei. While the primary thalamic nuclei are well understood to relay sensory information from the periphery to the cortex, the role of secondary sensory nuclei is elusive. We trained head-fixed mice to attend to one sensory modality while ignoring a second modality, namely to attend to touch and ignore vision, or vice versa. Arrays were used to record simultaneously from the secondary somatosensory thalamus (POm) and secondary visual thalamus (LP). In mice trained to respond to tactile stimuli and ignore visual stimuli, POm was robustly activated by touch and largely unresponsive to visual stimuli. A different pattern was observed when mice were trained to respond to visual stimuli and ignore touch, with POm now more robustly activated during visual trials. This POm activity was not explained by differences in movements (i.e. whisking, licking, pupil dilation) resulting from the two tasks. Post hoc histological reconstruction of array tracks through POm revealed that subregions varied in their degree of plasticity. LP exhibited similar phenomena. We conclude that behavioral training reshapes activity in secondary thalamic nuclei. Secondary nuclei respond to the same behaviorally relevant, reward-predicting stimuli regardless of stimulus modality.

Overview of 3D Printing Multimodal Flexible Sensors.

With the growing demand for flexible sensing systems and precision engineering, there is an increasing need for sensors that can accurately measure and analyze multimode signals. 3D printing technology has emerged as a crucial tool in the development of multimodal flexible sensors due to its advantages in design flexibility and manufacturing complex structures. This paper provides a review of recent advancements in 3D printing technology within the field of multimode flexible sensors, with particular emphasis on the relevant working mechanisms involved in decoupling complex signals. First, the research status of 3D printed multimodal flexible sensors is discussed, including their responsiveness to different modal stimuli such as mechanics, temperature, and gas. Furthermore, it explores methods for decoupling multimodal signals through structural and material design, artificial intelligence, and other technologies. Finally, this paper summarizes current challenges such as limited material selection, difficulties in miniaturization integration, and crosstalk between multisignal outputs. It also looks forward to future research directions in this area.

Verbal stimuli allow for symmetrical S-R priming effects between size and space.

The spatial-size association of response codes (SSARC) effect refers to the observation that left responses are faster and more accurate to small stimuli whereas right responses are faster and more accurate to large stimuli, as compared to the reverse assignment. The underlying spatial-size associations are strongly asymmetrical with physical size/location stimuli and vocal location/size responses and allow for regular but not reciprocal SSARC effects. Recent evidence, however, points towards an important role of stimulus mode in the emergence of reciprocal compatibility effects. We investigated the reciprocity of the SSARC effect with a different stimulus mode, namely with verbal size/location stimuli and vocal responses. In a size-location task, participants vocally responded to the words "small" or "large" by saying "left" or "right" according to a compatible ("small"-"left"/"large"-"right") or an incompatible mapping ("small"-"right"/"large"-"left"). In a location-size task, participants vocally responded to the words "left" or "right" by saying "small" or "large" according to a compatible ("left"-"small"/"right"-"large") or an incompatible ("left"-"large"/"right"-"small") mapping. We observed a regular and a reciprocal SSARC effect of similar size indicating symmetrical spatial-size associations. While regular SSARC effects thus emerge with verbal and physical size stimuli, reciprocal SSARC effects only emerge with verbal but not with physical location stimuli and vocal responses. Theoretical accounts of the SSARC effect differ in whether they predict reciprocal effects and whether they can account for the effect of stimulus mode on the reciprocal SSARC effect.

Attentional modulation of secondary somatosensory and visual thalamus of mice.

Each sensory modality has its own primary and secondary thalamic nuclei. While the primary thalamic nuclei are well understood to relay sensory information from the periphery to the cortex, the role of secondary sensory nuclei is elusive. We trained head-fixed mice to ateend to one sensory modality while ignoring a second modality, namely to ateend to touch and ignore vision, or vice versa. Arrays were used to record simultaneously from secondary somatosensory thalamus (POm) and secondary visual thalamus (LP). In mice trained to respond to tactile stimuli and ignore visual stimuli, POm was robustly activated by touch and largely unresponsive to visual stimuli. A different pateern was observed when mice were trained to respond to visual stimuli and ignore touch, with POm now more robustly activated during visual trials. This POm activity was not explained by differences in movements (i.e., whisking, licking, pupil dilation) resulting from the two tasks. Post hoc histological reconstruction of array tracks through POm revealed that subregions varied in their degree of plasticity. LP exhibited similar phenomena. We conclude that behavioral training reshapes activity in secondary thalamic nuclei. Secondary nuclei respond to the same behaviorally relevant, reward-predicting stimuli regardless of stimulus modality.

DISTINCT FUNCTIONAL CEREBRAL HYPERSENSITIVITY NETWORKS DURING INCISIONAL AND INFLAMMATORY PAIN IN RATS

Although the pathophysiology of pain has been investigated tremendously, there are still many open questions with regard to specific pain entities and their pain-related symptoms. To increase the translational impact of (preclinical) animal neuroimaging pain studies, the use of disease-specific pain models, as well as relevant stimulus modalities, are critical. We developed a comprehensive framework for brain network analysis combining functional magnetic resonance imaging (MRI) with graph-theory (GT) and data classification by linear discriminant analysis. This enabled us to expand our knowledge of stimulus modalities processing under incisional (INC) and pathogen-induced inflammatory (CFA) pain entities compared to acute pain conditions. GT-analysis has uncovered specific features in pain modality processing that align well with those previously identified in humans. These include areas such as S1, M1, CPu, HC, piriform, and cingulate cortex. Additionally, we have identified unique Network Signatures of Pain Hypersensitivity (NSPH) for INC and CFA. This leads to a diminished ability to differentiate between stimulus modalities in both pain models compared to control conditions, while also enhancing aversion processing and descending pain modulation. Our findings further show that different pain entities modulate sensory input through distinct NSPHs. These neuroimaging signatures are an important step toward identifying novel cerebral pain biomarkers for certain diseases and relevant outcomes to evaluate target engagement of novel therapeutic and diagnostic options, which ultimately can be translated to the clinic.

Control energy detects discrepancies in good vs. poor readers' structural-functional coupling during a rhyming task

Neuroimaging studies have identified functional and structural brain circuits that support reading. However, much less is known about how reading-related functional dynamics are constrained by white matter structure. Network control theory proposes that cortical brain dynamics are linearly determined by the white matter connectome, using control energy to evaluate the difficulty of the transition from one cognitive state to another. Here we apply this approach to linking brain dynamics with reading ability and disability in school-age children. A total of 51 children ages 8.25 -14.6 years performed an in-scanner rhyming task in visual and auditory modalities, with orthographic (spelling) and phonological (rhyming) similarity manipulated across trials. White matter structure and fMRI activation were used conjointly to compute the control energy of the reading network in each condition relative to a null fixation state. We then tested differences in control energy across trial types, finding higher control energy during non-word trials than word trials, and during incongruent trials than congruent trials. ROI analyses further showed a dissociation between control energy of the left fusiform and superior temporal gyrus depending on stimulus modality, with higher control energy for visual modalities in fusiform and higher control energy for auditory modalities in STG. Together, this study highlights that control theory can explain variations on cognitive demands in higher-level abilities such as reading, beyond what can be inferred from either functional or structural MRI measures alone.

Impact of Audio and Video Narratives Describing Financial Successes and Failures on Risk-Taking in the Balloon Analogue Risk Task

Narratives can influence risk-taking by shaping perceptions and emotions. In the present study we used first-person audio and video narratives describing either financial successes, failures, or neutral events devoid of any risk references in order to investigate how these types of stimuli affect risk-taking performance in the Balloon Analogue Risk Task (BART). Audio narratives of different types did not significantly alter risk-taking in the BART. However, videos depicting neutral narratives and narratives about financial successes surprisingly had risk-demoting influence compared to videos about financial failures, indexed by the percentage of popped balloons. The underlying reason for this finding remains to be understood. Additionally, gender differences emerged by the adjusted pumps measure in interaction with stimuli condition. Specifically, female participants displayed riskier tendencies after audio narratives, and conversely, video narratives led to a slight increase in risk-taking among male participants. These results emphasize the significant role of stimuli modality and indexing measures in risk-related studies.

Feedback inhibition by a descending GABAergic neuron regulates timing of escape behavior in Drosophila larvae

Escape behaviors help animals avoid harm from predators and other threats in the environment. Successful escape relies on integrating information from multiple stimulus modalities (of external or internal origin) to compute trajectories toward safe locations, choose between actions that satisfy competing motivations, and execute other strategies that ensure survival. To this end, escape behaviors must be adaptive. When a Drosophila melanogaster larva encounters a noxious stimulus, such as the focal pressure a parasitic wasp applies to the larval cuticle via its ovipositor, it initiates a characteristic escape response. The escape sequence consists of an initial abrupt bending, lateral rolling, and finally rapid crawling. Previous work has shown that the detection of noxious stimuli primarily relies on class IV multi-dendritic arborization neurons (Class IV neurons) located beneath the body wall, and more recent studies have identified several important components in the nociceptive neural circuitry involved in rolling. However, the neural mechanisms that underlie the rolling-escape sequence remain unclear. Here, we present both functional and anatomical evidence suggesting that bilateral descending neurons within the subesophageal zone of D. melanogaster larva play a crucial role in regulating the termination of rolling and subsequent transition to escape crawling. We demonstrate that these descending neurons (designated SeIN128) are inhibitory and receive inputs from a second-order interneuron upstream (Basin-2) and an ascending neuron downstream of Basin-2 (A00c). Together with optogenetic experiments showing that co-activation of SeIN128 neurons and Basin-2 influence the temporal dynamics of rolling, our findings collectively suggest that the ensemble of SeIN128, Basin-2, and A00c neurons forms a GABAergic feedback loop onto Basin-2, which inhibits rolling and thereby facilitates the shift to escape crawling.

Feedback inhibition by a descending GABAergic neuron regulates timing of escape behavior in Drosophila larvae.

Escape behaviors help animals avoid harm from predators and other threats in the environment. Successful escape relies on integrating information from multiple stimulus modalities (of external or internal origin) to compute trajectories toward safe locations, choose between actions that satisfy competing motivations, and execute other strategies that ensure survival. To this end, escape behaviors must be adaptive. When a Drosophila melanogaster larva encounters a noxious stimulus, such as the focal pressure a parasitic wasp applies to the larval cuticle via its ovipositor, it initiates a characteristic escape response. The escape sequence consists of an initial abrupt bending, lateral rolling, and finally rapid crawling. Previous work has shown that the detection of noxious stimuli primarily relies on class IV multi-dendritic arborization neurons (Class IV neurons) located beneath the body wall, and more recent studies have identified several important components in the nociceptive neural circuitry involved in rolling. However, the neural mechanisms that underlie the rolling-escape sequence remain unclear. Here, we present both functional and anatomical evidence suggesting that bilateral descending neurons within the subesophageal zone of D. melanogaster larva play a crucial role in regulating the termination of rolling and subsequent transition to escape crawling. We demonstrate that these descending neurons (designated SeIN128) are inhibitory and receive inputs from a second-order interneuron upstream (Basin-2) and an ascending neuron downstream of Basin-2 (A00c). Together with optogenetic experiments showing that co-activation of SeIN128 neurons and Basin-2 influence the temporal dynamics of rolling, our findings collectively suggest that the ensemble of SeIN128, Basin-2, and A00c neurons forms a GABAergic feedback loop onto Basin-2, which inhibits rolling and thereby facilitates the shift to escape crawling.

Inhibitory effects of prepulse stimuli on the electrophysiological responses to startle stimuli in the deep layers of the superior colliculus.

Prepulse inhibition (PPI) is a phenomenon where a weak prepulse stimulus inhibits the startle reflex to a subsequent stronger stimulus, which can be induced by various sensory stimulus modalities such as visual, tactile, and auditory stimuli. This study investigates the neural mechanisms underlying auditory PPI by focusing on the deep layers of the superior colliculus (deepSC) and the inferior colliculus (IC) in rats. Nineteen male Sprague-Dawley rats were implanted with electrodes in the left deepSC and the right IC, and electrophysiological recordings were conducted under anesthesia to observe the frequency following responses (FFRs) to startle stimuli with and without prepulse stimuli. Our results showed that in the deepSC, narrowband noise as a prepulse stimulus significantly inhibited the envelope component of the startle response, while the fine structure component remained unaffected. However, this inhibitory effect was not observed in the IC or when the prepulse stimulus was a gap. These findings suggest that the deepSC plays a crucial role in the neural circuitry of PPI, particularly in the modulation of the envelope component of the startle response. The differential effects of narrowband noise and gap as prepulse stimuli also indicate distinct neural pathways for sound-induced PPI and Gap-PPI. Understanding these mechanisms could provide insights into sensory processing and potential therapeutic targets for disorders involving impaired PPI, such as tinnitus.

Erroneous saccade co-execution during manual action control is independent of oculomotor stimulus-response translation ease

Recent multiple action control studies have demonstrated difficulties with single-action (vs. dual-action) execution when accompanied by the requirement to inhibit a prepotent additional response (e.g., a highly automatic eye movement). Such a dual-action performance benefit is typically characterized by frequent false-positive executions of the currently unwarranted response. Here, we investigated whether the frequency of false-positive saccades is affected by the ease of translating a stimulus into a spatial oculomotor response (S-R translation ease): Is it harder to inhibit a saccade that is more automatically triggered via the stimulus? Participants switched on a trial-by-trial basis between executing a single saccade, a single manual button press, and a saccadic-manual dual action in response to a single visual stimulus. Importantly, we employed three different stimulus modes that varied in oculomotor S-R translation ease (peripheral square > central arrow > central shape). The hierarchy of S-R translation ease was reflected by increasing saccade and manual reaction times. Critically, however, the frequency of false-positive saccades in single manual trials was not substantially affected by the stimulus mode. Our results rule out explanations related to limited capacity sharing (between inhibitory control and S-R translation demands) as well as accounts related to the time available for the completion of saccade inhibition. Instead, the findings suggest that the erroneous co-activation of the oculomotor system was elicited by the mere execution of a (frequently associated) manual response (action-based co-activation).