Multimodal Sensory Information Research Articles

Rat brains also have a default mode network

The default mode network (DMN) in humans has been suggested to support a variety of cognitive functions and has been implicated in an array of neuropsychological disorders. However, its function(s) remains poorly understood. We show that rats possess a DMN that is broadly similar to the DMNs of nonhuman primates and humans. Our data suggest that, despite the distinct evolutionary paths between rodent and primate brain, a well-organized, intrinsically coherent DMN appears to be a fundamental feature in the mammalian brain whose primary functions might be to integrate multimodal sensory and affective information to guide behavior in anticipation of changing environmental contingencies.

Synthesis of Distributed Cognitive Systems: Interacting Maps for Sensor Fusion

Event Abstract Back to Event Synthesis of Distributed Cognitive Systems: Interacting Maps for Sensor Fusion Cristian Axenie1* and Jorg Conradt1 1 Technische Universität München, Electrical Engineering and Information Technology, Germany Environmental interaction is an important aspect in the life of every physical entity, which impacts its internal state and allows the acquisition of new behaviors. A prerequisite for such interaction is multimodal sensory fusion with the goal of providing a consistent internal representation of the external world. In technical systems, such a representation is obtained by fusing noisy data from various available sensors, which combined generate a richer representation compared to individual use. State-of-the-art technologies apply probabilistic methods for combining prior and observed information, but these techniques are computationally inefficient for a large number of sensors. In our project we propose extending an existing initial model of interacting maps for sensory interpretation with the focus on multimodal sensory fusion. The core aspect here is to design and implement a neuro-biologically inspired method for real-time distributed interpretation of sensory stimuli in a mobile robotic setting: a distributed graphical computing system with inter-merged information storage and processing that allows efficient parallel reasoning. This network architecture consists of interconnected heterogeneous software units, each encoding and processing a different feature about the environment that is represented in a local map. A map here is defined as a pixel based representation of some perceived or inferred aspect of the current environment, in which each entry contains a value of a certain type (e.g. a scalar of image brightness or a 2D vector of local optic flow). Such extracted pieces of environmental knowledge interact by mutual influence to ensure overall system coherence: the values represented in each map slowly drift in the direction that minimizes disagreement between the two sides of an assumed inter-map relationship through mutual influence. Although by system design each map specializes in representing a certain feature of the environment, the required relations between maps for such influence are established through a learning process: the true relations between maps are a priori unknown, but exist hidden within the observed real-world data. Therefore, our system needs to infer those relations over time from correlated sensory signals. Starting from the proposed distributed processing architecture, our main research investigates self-constructing and adapting internal relations between maps (structural plasticity) that reflect the underlying relations between multimodal sensory information streams. We investigate optimized network distributions on available computing resources without an explicit a priori network topology provided by a systems designer. This new architecture reflects distributed information processing as known to occur in brains, and ensures a fast, robust and scalable computational architecture appropriate for real-time real-world robotic applications. Acknowledgements The foundations of this work have been laid during workshops such as the Telluride and the CapoCaccia Cognitive Neuromorphic Engineering Workshop. Funding and inspiration for application scenarios has come from the DFG Cluster of Excellence “CoTeSys” and the Muinch Bernstein Center for Computational Neuroscience. Keywords: network computation, self-growing systems, sensory fusion Conference: Bernstein Conference 2012, Munich, Germany, 12 Sep - 14 Sep, 2012. Presentation Type: Poster Topic: Sensory processing and perception Citation: Axenie C and Conradt J (2012). Synthesis of Distributed Cognitive Systems: Interacting Maps for Sensor Fusion. Front. Comput. Neurosci. Conference Abstract: Bernstein Conference 2012. doi: 10.3389/conf.fncom.2012.55.00077 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 22 May 2012; Published Online: 12 Sep 2012. * Correspondence: Mr. Cristian Axenie, Technische Universität München, Electrical Engineering and Information Technology, Munich, 80333, Germany, cristian.axenie@tum.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Cristian Axenie Jorg Conradt Google Cristian Axenie Jorg Conradt Google Scholar Cristian Axenie Jorg Conradt PubMed Cristian Axenie Jorg Conradt Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

A Neuromorphic VLSI Head Direction Cell System

The head direction (HD) cell system in the brain of mammals is thought to be part of the neural circuitry supporting their spatial navigation capabilities. In this paper we present a neuromorphic very large scale integration system that models the operation of the HD system. Relying on spiking neurons and attractor dynamics, this system can be used to represent and retain the current estimate of head orientation in the environment and integrate the angular velocity to update this estimate. The presented system is more than a simple integrator and orientation memory; it is a first step towards modeling sophisticated biological navigation systems that use multimodal sensory and motor information to continually realign themselves based on experience.

Changing and shielded magnetic fields suppress c-Fos expression in the navigation circuit: input from the magnetosensory system contributes to the internal representation of space in a subterranean rodent

The neural substrate subserving magnetoreception and magnetic orientation in mammals is largely unknown. Previous experiments have demonstrated that the processing of magnetic sensory information takes place in the superior colliculus. Here, the effects of magnetic field conditions on neuronal activity in the rodent navigation circuit were assessed by quantifying c-Fos expression. Ansell's mole-rats (Fukomys anselli), a mammalian model to study the mechanisms of magnetic compass orientation, were subjected to natural, periodically changing, and shielded magnetic fields while exploring an unfamiliar circular arena. In the undisturbed local geomagnetic field, the exploration of the novel environment and/or nesting behaviour induced c-Fos expression throughout the head direction system and the entorhinal-hippocampal spatial representation system. This induction was significantly suppressed by exposure to periodically changing and/or shielded magnetic fields; discrete decreases in c-Fos were seen in the dorsal tegmental nucleus, the anterodorsal and the laterodorsal thalamic nuclei, the postsubiculum, the retrosplenial and entorhinal cortices, and the hippocampus. Moreover, in inactive animals, magnetic field intensity manipulation suppressed c-Fos expression in the CA1 and CA3 fields of the hippocampus and the dorsal subiculum, but induced expression in the polymorph layer of the dentate gyrus. These findings suggest that key constituents of the rodent navigation circuit contain populations of neurons responsive to magnetic stimuli. Thus, magnetic information may be integrated with multimodal sensory and motor information into a common spatial representation of allocentric space within this circuit.

Flexible responses to visual and olfactory stimuli by foragingManduca sexta: larval nutrition affects adult behaviour

Here, we show that the consequences of deficient micronutrient (beta-carotene) intake during larval stages of Manduca sexta are carried across metamorphosis, affecting adult behaviour. Our manipulation of larval diet allowed us to examine how developmental plasticity impacts the interplay between visual and olfactory inputs on adult foraging behaviour. Larvae of M. sexta were reared on natural (Nicotiana tabacum) and artificial laboratory diets containing different concentrations of beta-carotene (standard diet, low beta-carotene, high beta-carotene and cornmeal). This vitamin-A precursor has been shown to be crucial for photoreception sensitivity in the retina of M. sexta. After completing development, post-metamorphosis, starved adults were presented with artificial feeders that could be either scented or unscented. Regardless of their larval diet, adult moths fed with relatively high probabilities on scented feeders. When feeders were unscented, moths reared on tobacco were more responsive than moths reared on beta-carotene-deficient artificial diets. Strikingly, moths reared on artificial diets supplemented with increasing amounts of beta-carotene (low beta and high beta) showed increasing probabilities of response to scentless feeders. We discuss these results in relationship to the use of complex, multi-modal sensory information by foraging animals.

Multimodal sensory information requirements for enhancing situation awareness and training effectiveness

Virtual training systems use multimodal technology to provide realistic training scenarios. To determine the benefits of adopting multimodal training strategies, it is essential to identify the critical knowledge, skills and attitudes that are being targeted in training and relate these to the multimodal human sensory systems that should be stimulated to support this acquisition. This paper focuses on trainee situation awareness and develops a multimodal optimisation of situation awareness conceptual model that outlines how multimodality may be used to optimise perception, comprehension and prediction of object recognition, spatial and temporal components of awareness. Specific multimodal design techniques are presented, which map desired training outcomes and supporting sensory requirements to training system design guidelines for optimal trainee situation awareness and human performance.

`Songese': maternal structuring of musical interaction with infants

The aim of this study was to investigate the temporal structure of mother—infant interactions with songs, with particular attention to two aspects: 1) the singing of the mothers to their infants, and 2) the non-verbal behaviours mothers and infants produce in synchrony with the musical beat. Four mother—infant dyads were video-recorded when the infants were 3—4 months of age and again when they were 7—8 months old. Mothers were asked to sing songs of their choice while interacting with their infants. Analyses of the mothers' singing and synchronous behaviours with the beat revealed that they emphasized the hierarchical structure of the song and provided a segmentation of the temporal structure of the interaction. Infants were shown to be sensitive to their mothers' emphasis by producing significantly more synchronous behaviours on some beats than on others. It is suggested that the multimodal sensory information provided by the mothers ensures the infants' effective learning and conveys the characteristics of what could be called `songese' — the equivalent of `motherese'.

The role of olfaction and vision in the foraging behaviour of an echolocating megachiropteran fruit bat, Rousettus leschenaulti (Pteropodidae)

Many mammals use multimodal sensory information to find their food in complex environments. We studied the roles of olfactory and visual cues in the foraging behaviour of Rousettus leschenaulti, a tongue-clicking megachiropteran bat. We conducted experiments by offering a whole fruit, mashed fruit without shape and an artificial fruit to R. leschenaulti in dim light as well as in total darkness. R. leschenaulti responded to whole guava fruit and mashed guava fruit even in total darkness, whereas, artificial fruit similar to real fruit in size, shape and structure was ignored even when illuminated. These results reveal that odour may act as the primary cue for R. leschenaulti in finding fruit.

Courtship Initiation Is Stimulated by Acoustic Signals in Drosophila melanogaster

Finding a mating partner is a critical task for many organisms. It is in the interest of males to employ multiple sensory modalities to search for females. In Drosophila melanogaster, vision is thought to be the most important courtship stimulating cue at long distance, while chemosensory cues are used at relatively short distance. In this report, we show that when visual cues are not available, sounds produced by the female allow the male to detect her presence in a large arena. When the target female was artificially immobilized, the male spent a prolonged time searching before starting courtship. This delay in courtship initiation was completely rescued by playing either white noise or recorded fly movement sounds to the male, indicating that the acoustic and/or seismic stimulus produced by movement stimulates courtship initiation, most likely by increasing the general arousal state of the male. Mutant males expressing tetanus toxin (TNT) under the control of Gr68a-GAL4 had a defect in finding active females and a delay in courtship initiation in a large arena, but not in a small arena. Gr68a-GAL4 was found to be expressed pleiotropically not only in putative gustatory pheromone receptor neurons but also in mechanosensory neurons, suggesting that Gr68a-positive mechanosensory neurons, not gustatory neurons, provide motion detection necessary for courtship initiation. TNT/Gr68a males were capable of discriminating the copulation status and age of target females in courtship conditioning, indicating that female discrimination and formation of olfactory courtship memory are independent of the Gr68a-expressing neurons that subserve gustation and mechanosensation. This study suggests for the first time that mechanical signals generated by a female fly have a prominent effect on males' courtship in the dark and leads the way to studying how multimodal sensory information and arousal are integrated in behavioral decision making.

Neurology of musical performance

Performing music at a professional level requires the integration of multimodal sensory and motor information and precise monitoring of the performance via auditory feedback. In the context of Western classical music, musicians are forced to reproduce highly controlled movements almost perfectly with a high reliability. These specialised sensorimotor skills are acquired during extensive training periods over many years. The superior skills of musicians are mirrored in functional and structural plastic adaptations of sensorimotor and auditory systems of the brain. Auditory-sensorimotor integration, for example, is accompanied by rapid modulations of neuronal connectivity in the time range of 20 minutes. Finally, dysfunctional plasticity in musicians, known as musician's dystonia, leads to deterioration of extensively trained fine motor skills. Musician's dystonia may be caused by training induced dysplasticity with pathological fusion of central nervous representations in sensorimotor cortical and subcortical brain regions.

Insular cortex and neuropsychiatric disorders: A review of recent literature

The insular cortex is located in the centre of the cerebral hemisphere, having connections with the primary and secondary somatosensory areas, anterior cingulate cortex, amygdaloid body, prefrontal cortex, superior temporal gyrus, temporal pole, orbitofrontal cortex, frontal and parietal opercula, primary and association auditory cortices, visual association cortex, olfactory bulb, hippocampus, entorhinal cortex, and motor cortex. Accordingly, dense connections exist among insular cortex neurons. The insular cortex is involved in the processing of visceral sensory, visceral motor, vestibular, attention, pain, emotion, verbal, motor information, inputs related to music and eating, in addition to gustatory, olfactory, visual, auditory, and tactile data. In this article, the literature on the relationship between the insular cortex and neuropsychiatric disorders was summarized following a computer search of the Pub-Med database. Recent neuroimaging data, including voxel based morphometry, PET and fMRI, revealed that the insular cortex was involved in various neuropsychiatric diseases such as mood disorders, panic disorders, PTSD, obsessive-compulsive disorders, eating disorders, and schizophrenia. Investigations of functions and connections of the insular cortex suggest that sensory information including gustatory, olfactory, visual, auditory, and tactile inputs converge on the insular cortex, and that these multimodal sensory information may be integrated there.

A reexamination of the size–weight illusion induced by visual size cues

The size-weight illusion induced by visually perceived sizes was reexamined to investigate whether this illusion is a sensory based or cognitive-based phenomenon. A computer-augmented environment was utilized to manipulate visual size information of target objects independently of their haptic information. Two physical cubes of equal mass (30.0 g) and size (3.0 x 3.0 x 3.0 cm) were suspended in parallel by wires attached to small graspable rings, in order to keep haptically obtained information constant between lifts. Instead of directly seeing each physical cube, subjects viewed 3D graphics of a cube with a wire and a ring that were precisely superimposed onto each physical cube. Seventeen subjects vertically lifted these augmented cubes, one after the other, by grasping the attached rings, and then reported their perception of cube heaviness. The graphical size of a comparison cube pseudo randomly varied for every comparison from 1.0 x 1.0 x 1.0 to 9.0 x 9.0 x 9.0 cm, while that of a standard cube remained constant (5.0 x 5.0 x 5.0 cm). Results indicated that the size-weight illusion frequently and systematically occurred for all the subjects such that when the comparison cube was relatively smaller than the standard cube, it was perceived to be heavier, and vice versa. As the size difference increased between the standard cube and the comparison cube, more subjects experienced the illusion, and vice versa. Follow-up tests showed occurrence of the size-weight illusion was significantly correlated with subject's sensitivity to discriminate weight, but not with sensitivity to discriminate visual size. Results suggest that the size-weight illusion induced by only visual size cues in an augmented environment is sensory based, and depends on an individual's integrated perception based on multimodal sensory information.

Somatosensory-Motor Neuronal Activity in the Superior Colliculus of the Primate

The superior colliculus (SC) in primates plays an important role in orienting gaze and arms toward novel stimuli. Here we ask whether neurons in the intermediate and deep layers of the SC are also involved in the interaction with objects. In two trained monkeys we found a large number of SC units that were specifically activated when the monkeys contacted and pushed a target that had been reached with either hand. These neurons, however, were silent when the monkeys simply looked at or reached for the target but did not touch it. The activity related to interacting with objects was spatially tuned and increased with push strength. Neurons in the SC with this type of activity may be involved in a somatosensory-motor feedback loop that monitors the force of the active muscles together with the spatial position of the limb required for proper interaction with an object.

Brain Neuroimaging in Cannabis Use: A Review

In this study, the authors systematically reviewed structural and functional neuroimaging studies of cannabis use. Structural abnormalities generally have not been identified with chronic use. Regular users demonstrate reciprocal changes in brain activity globally and in cerebellar and frontal regions. Abstinence results in decreases, and administration results in increases correlating with subjective intoxication. Chronic use and cannabis administration result in attenuated brain activity in task-activated regions or activation of compensatory regions. Findings correlate partially with neuropsychological data, but generalization is limited by the lack of use of diagnostic criteria, appropriately paired neuropsychological testing or means to better quantify cannabis use and abstinence.

Analyzing a complex visuomotor tracking task with brain-electrical event related potentials

Non-invasive techniques such as neuroimaging and event-related potential (ERP) methods have dramatically enhanced our understanding of the human brain. According to the requirements of the applied method, it is useful to simplify tasks for methodological reasons. In the present study we tested whether ERP measures are also suitable for analyzing complex tasks. In order to do this, we developed an analysis strategy based on the post hoc analysis of the behavioural data. We applied this method to a pursuit-tracking task of 25 s trial duration, consisting of repeated and non-repeated waveforms, where subjects had to track a target cross with a mouse-controlled cursor cross. An EEG was recorded from 62 channels. Response-locked ERPs were computed for two types of error correction: the correction of errors induced externally by the change of target direction and of internal errors generated by the subject itself. We found several ERP components that could be assigned to different feedback and feedforward controlled processing steps in the frontoparietal circuitry underlying visuomotor control, such as movement planning, movement execution (motor potential), reafferent activity, visuospatial analysis, and attentional (P300) processes. Our results support newer models that propose a role for the posterior parietal cortex in integrating multimodal sensory information. In addition, fast (about 180 ms and probably facilitated by anticipation) and slow (about 230–260 ms) error corrections could be differentiated by the time course of ERP activity. Our results show that complex (motor) tasks can be investigated with ERPs. This opens fruitful perspectives for future research on motor control in an ecological setting.

New evidence for involvement of the entorhinal region in schizophrenia: a combined MRI volumetric and DTI study

Postmortem examinations and magnetic resonance imaging (MRI) studies suggest involvement of the entorhinal cortex (EC) in schizophrenic psychoses. However, the extent and nature of the possible pathogenetical process underlying the observed alterations of this limbic key region for processing of multimodal sensory information remains unclear. Three-dimensional high-resolution MRI volumetry and evaluation of the regional diffusional anisotropy based on diffusion tensor imaging (DTI) were performed on the EC of 15 paranoid schizophrenic patients and 15 closely matched control subjects. In schizophrenic patients, EC volumes showed a slight, but not significant, decrease. However, the anisotropy values, expressed as inter-voxel coherences (COH), were found to be significantly decreased by 17.9% (right side) and 12.5% (left side), respectively, in schizophrenics. Reduction of entorhinal diffusional anisotropy can be hypothesized to be functionally related to disturbances in the perforant path, the principal efferent EC fiber tract supplying the limbic system with neuronal input from multimodal association centers. Combinations of different MRI modalities are a promising approach for the detection and characterization of subtle brain tissue alterations.

Human medial intraparietal cortex subserves visuomotor coordinate transformation

In the macaque, the posterior parietal cortex (PPC) integrates multimodal sensory information for planning and coordinating complex movements. In particular, the areas around the intraparietal sulcus (IPS) serve as an interface between the sensory and motor systems to allow for coordinated movements in space. Because recent imaging studies suggest a comparable functional and anatomical organization of human and monkey IPS, we hypothesized that in humans, as in macaques, the medial intraparietal cortex (area MIP) subserves visuomotor transformations. To test this hypothesis, changes of neural activity were measured using functional magnetic resonance imaging (fMRI) while healthy subjects performed a joystick paradigm similar to the ones previously employed in macaques for studying area MIP. As hypothesized, visuomotor coordinate transformation subserving goal-directed hand movements activated superior parietal cortex with the local maximum of increased neural activity lying in the medial wall of IPS. Compared to the respective visuomotor control conditions, goal-directed hand movements under predominantly proprioceptive control activated a more anterior part of medial IPS, whereas posterior medial IPS was more responsive to visually guided hand movements. Contrasting the two coordinate transformation conditions, changing the modality of movement guidance (visual/proprioceptive) did not significantly alter the BOLD signal within IPS but demonstrated differential recruitment of modality specific areas such as V5/MT and sensorimotor cortex/area 5, respectively. The data suggest that the human medial intraparietal cortex subserves visuomotor transformation processes to control goal-directed hand movements independently from the modality-specific processing of visual or proprioceptive information.

Sensorimotor Integration in Basal Ganglia Disorders: New Findings on Impaired Kinaesthesia in Focal Dystonia and Parkinson's Disease

The term „movement disorders“ is often used to subsume conditions resulting from basal ganglia pathology and refers to the predominating motor symptoms. However, clinically unapparent sensory deficits were also found in these disorders. Recent neurophysiological, neuroimaging and animal studies have revealed that sensory processing is impaired in Huntington's disease, Parkinson's disease and focal dystonia. This raised the interesting question to what extent impaired sensorimotor integration contributes to the clinical symptoms in these disorders. We analyzed kinaesthesia, the conscious perception of limb position, in a passive movement task (repeated extension and flexion of the index finger between 0.2° and 4° in 45 trials of random order) and compared results of patients with idiopathic focal dystonia (blepharospasm n=3, torticollis n=6) and patients with idiopathic PD (n=8) to an age-matched group of healthy controls (n=5). Patients and controls were without cognitive decline and did not show sensory deficits in routine clinical examination. PD patients detected the correct direction of a 0.2° displacement of the index finger in only 59% of the trials compared to 82% in healthy controls. Patients with focal dystonia also revealed a kinaesthetic deficit given that they responded correctly in only 76% of trials in the same task. The overall differences for detecting the correct direction of index finger movements between patient groups and controls were statistically significant (β2, p=0.001 for PD group, p=0.002 for dystonia group). Calculated thresholds for 75% correct responses were 0.3° in PD patients, 0.2° in dystonia patients and 0.1° in control subjects. Additionally, we analyzed the ability to judge the extent of a passive movement of the index finger by asking the patients to compare two consecutive passive movements. PD patients appeared to be more impaired than dystonia patients and both groups were different from the control group. Our findings in PD patients confirm results of previous studies that revealed kinaesthetic deficits in PD. Furthermore, the results of the present study suggest that a kinaesthetic impairment is also prevalent in focal dystonia. Together with previous findings the present results indicate that a deficit in processing multimodal sensory information might contribute to motor deficits in distinct basal ganglia disorders. Our results show the potential role of the basal ganglia as a sensory analyzer.

A generic neural network for multi-modal sensorimotor learning

A generic neural network module has been developed, which learns to combine multi-modal sensory information to produce adequate motor commands. The module learns in the first step to combine multi-modal sensory information, based on which it subsequently learns to control a kinematic arm. The module can learn to combine two sensory inputs whatever their modality. We report the architecture and learning strategy of the module and characterize its performance by simulations in two situations of reaching by a linear arm with multiple degrees of freedom: (1) mapping of tactile and arm-related proprioceptive information, and (2) mapping of gaze and arm-related proprioceptive information.

Looking to improve haptic performance

Human dexterity belies the complexity of processing required for accurate hand–eye coordination. Likewise, the seamless, holistic percept of the outside world that we enjoy belies the complicated integration of multi-modal sensory information necessarily occurring in our brains. For example, it's been shown that perceptual performance improves cumulatively as more senses are concentrated to the same region of space. But for optimal interaction with the environment, our senses need to communicate not just with each other but also with systems responsible for our movement control.