Shared Representation Research Articles

Heterogeneous Sensor Data Fusion By Deep Multimodal Encoding

Heterogeneous sensor data fusion is a challenging field that has gathered significant interest in recent years. Two of these challenges are learning from data with missing values, and finding shared representations for multimodal data to improve inference and prediction. In this paper, we propose a multimodal data fusion framework, the deep multimodal encoder (DME), based on deep learning techniques for sensor data compression, missing data imputation, and new modality prediction under multimodal scenarios. While traditional methods capture only the intramodal correlations, DME is able to mine both the intramodal correlations in the initial layers and the enhanced intermodal correlations in the deeper layers. In this way, the statistical structure of sensor data may be better exploited for data compression. By incorporating our new objective function, DME shows remarkable ability for missing data imputation tasks in sensor data. The shared multimodal representation learned by DME may be used directly for predicting new modalities. In experiments with a real-world dataset collected from a 40-node agriculture sensor network which contains three modalities, DME can achieve a root mean square error (RMSE) of missing data imputation which is only 20% of the traditional methods like K -nearest neighbors and sparse principal component analysis and the performance is robust to different missing rates. It can also reconstruct temperature modality from humidity and illuminance with an RMSE of $7\; {}^{\circ }$ C, directly from a highly compressed (2.1%) shared representation that was learned from incomplete (80% missing) data.

Multimodal shared features learning for emotion recognition by enhanced sparse local discriminative canonical correlation analysis

Multimodal emotion recognition is a challenging research topic which has recently started to attract the attention of the research community. To better recognize the video users’ emotion, the research of multimodal emotion recognition based on audio and video is essential. Multimodal emotion recognition performance heavily depends on finding good shared feature representation. The good shared representation needs to consider two aspects: (1) it has the character of each modality and (2) it can balance the effect of different modalities to make the decision optimal. In the light of these, we propose a novel Enhanced Sparse Local Discriminative Canonical Correlation Analysis approach (En-SLDCCA) to learn the multimodal shared feature representation. The shared feature representation learning involves two stages. In the first stage, we pretrain the Sparse Auto-Encoder with unimodal video (or audio), so that we can obtain the hidden feature representation of video and audio separately. In the second stage, we obtain the correlation coefficients of video and audio using our En-SLDCCA approach, then we form the shared feature representation which fuses the features from video and audio using the correlation coefficients. We evaluate the performance of our method on the challenging multimodal Enterface’05 database. Experimental results reveal that our method is superior to the unimodal video (or audio) and improves significantly the performance for multimodal emotion recognition when compared with the current state of the art.

Embodied emotion impairment in Huntington's Disease

Theories of embodied cognition suggest that perceiving an emotion involves somatovisceral and motoric re-experiencing. Here we suggest taking such an embodied stance when looking at emotion processing deficits in patients with Huntington's Disease (HD), a neurodegenerative motor disorder. The literature on these patients' emotion recognition deficit has recently been enriched by some reports of impaired emotion expression. The goal of the study was to find out if expression deficits might be linked to a more motoric level of impairment. We used electromyography (EMG) to compare voluntary emotion expression from words to emotion imitation from static face images, and spontaneous emotion mimicry in 28 HD patients and 24 matched controls. For the latter two imitation conditions, an underlying emotion understanding is not imperative (even though performance might be helped by it). EMG measures were compared to emotion recognition and to the capacity to identify and describe emotions using alexithymia questionnaires. Alexithymia questionnaires tap into the more somato-visceral or interoceptive aspects of emotion perception. Furthermore, we correlated patients' expression and recognition scores to cerebral grey matter volume using voxel-based morphometry (VBM). EMG results replicated impaired voluntary emotion expression in HD. Critically, voluntary imitation and spontaneous mimicry were equally impaired and correlated with impaired recognition. By contrast, alexithymia scores were normal, suggesting that emotion representations on the level of internal experience might be spared. Recognition correlated with brain volume in the caudate as well as in areas previously associated with shared action representations, namely somatosensory, posterior parietal, posterior superior temporal sulcus (pSTS) and subcentral sulcus. Together, these findings indicate that in these patients emotion deficits might be tied to the “motoric level” of emotion expression. Such a double-sided recognition and expression impairment may have important consequences, interrupting empathy in nonverbal communication both ways (understanding and being understood), independently of intact internal experience of emotion.

Co-evolutionary multi-task learning with predictive recurrence for multi-step chaotic time series prediction

Multi-task learning employs a shared representation of knowledge for learning several instances of the same problem. Multi-step time series problem is one of the most challenging problems for machine learning methods. The performance of a prediction model face challenges for higher prediction horizons due to the accumulation of errors. Cooperative coevolution employs in a divide and conquer approach for training neural networks and has been very promising for single step ahead time series prediction. Recently, co-evolutionary multi-task learning has been proposed for dynamic time series prediction. In this paper, we adapt co-evolutionary multi-task learning for multi-step prediction where predictive recurrence is developed to feature knowledge from previous states for future prediction horizon. The goal of the paper is to present a network architecture with predictive recurrence which is capable of multi-step prediction through a form of multi-task learning. We employ cooperative neuro-evolution and an evolutionary algorithm as baselines for comparison. The results show that the proposed method provides the best generalization performance in most cases. Comparison of results with the literature has shown to be promising which motivates further application of the approach for related real-world problems.

Deep Multimodal Fusion: A Hybrid Approach

We propose a novel hybrid model that exploits the strength of discriminative classifiers along with the representation power of generative models. Our focus is on detecting multimodal events in time varying sequences as well as generating missing data in any of the modalities. Discriminative classifiers have been shown to achieve higher performances than the corresponding generative likelihood-based classifiers. On the other hand, generative models learn a rich informative space which allows for data generation and joint feature representation that discriminative models lack. We propose a new model that jointly optimizes the representation space using a hybrid energy function. We employ a Restricted Boltzmann Machines (RBMs) based model to learn a shared representation across multiple modalities with time varying data. The Conditional RBMs (CRBMs) is an extension of the RBM model that takes into account short term temporal phenomena. The hybrid model involves augmenting CRBMs with a discriminative component for classification. For these purposes we propose a novel Multimodal Discriminative CRBMs (MMDCRBMs) model. First, we train the MMDCRBMs model using labeled data by training each modality, followed by training a fusion layer. Second, we exploit the generative capability of MMDCRBMs to activate the trained model so as to generate the lower-level data corresponding to the specific label that closely matches the actual input data. We evaluate our approach on ChaLearn dataset, audio-mocap, as well as the Tower Game dataset, mocap-mocap as well as three multimodal toy datasets. We report classification accuracy, generation accuracy, and localization accuracy and demonstrate its superiority compared to the state-of-the-art methods.

The Cognitive Estimation Task is nonunitary: Evidence for multiple magnitude representation mechanisms among normative and ADHD college students

There is a current debate on whether the cognitive system has a shared representation for all magnitudes or whether there are unique representations. To investigate this question, we used the Biber cognitive estimation task. In this task, participants were asked to provide estimates for questions such as, “How many sticks of spaghetti are in a package?” The task uses different estimation categories (e.g., time, numerical quantity, distance, and weight) to look at real-life magnitude representations. Experiment 1 assessed (N = 95) a Hebrew version of the Biber Cognitive Estimation Task and found that different estimation categories had different relations, for example, weight, time, and distance shared variance, but numerical estimation did not. We suggest that numerical estimation does not require the use of measurement in units, hence, it represents a more “pure” numerical estimation. Experiment 2 found that different factors explain individual abilities in different estimation categories. For example, numerical estimation was predicted by preverbal innate quantity understanding (approximate number sense) and working memory, whereas time estimations were supported by IQ. These results demonstrate that cognitive estimation is not a unified construct.

Expectations, Satisfaction and Professional Achievement. An Analysis of their Relation in light of a New Systemic Paradigm: The Three- Dimensional Spiral of Sense

This research has been done in the field of Education, Sociology and Social and Organizational Psychology. Issues are taken up from two research studies carried out with researchers and PhDs from different disciplinary fields and national contexts. The relationship between Expectations, levels of Satisfaction and Professional Achievement are analyzed in light of different theories: Expectancy-Valence, Attributional theories, Education theories such as Consumption or Investment / Human Capital and the theory of Anomy. The methodology utilized was quantitative-qualitative: two questionnaires, one semi-structured survey including open phrases which allowed actors to speak freely, one interview and, in the case of the French-Argentine research with PhDs, a special qualitative technique was applied: hierarchical evocation. This allowed us to determine which aspects related to professional pathways (objective and subjective) formed part of the “core” of social or shared representations and which were secondary aspects at the periphery of said core. Our findings show non-linear relationships between study variables – Expectations, Satisfaction and Achievement – and self-sustained interplay along three levels: micro individual, meso organizational and macro social. These are interpreted in light of a new systemic paradigm in human and social sciences, a paradigm which the author defines as “The Three-Dimensional Spiral of Sense”.

My cattle and your park: codesigning a role-playing game with rural communities to promote multistakeholder dialogue at the edge of protected areas

Protected areas are often burdened with conflicts between environmental managers and neighboring rural communities. Unsuccessful top-down approaches for conservation may be replaced by alternative forms of systemic management involving local actors in the design and implementation of conservation management. Although theoretically sound and appealing, the involvement of local residents in the management of protected areas is often impaired in practice by scale mismatches, conflicting values and interests, power imbalance, and a lack of trust among actors. In this paper, we describe a process initiated in Zimbabwe to create a fair and balanced locally designed arena where local communities and protected area managers may collaborate to produce effective management plans. Adopting the Companion Modeling approach, we conducted a participatory modeling experiment to codesign a role-playing game that simulates the interactions between farming activities, livestock herding practices, and wildlife in a virtual landscape reproducing local social–ecological dynamics. After 18 months of intensive ethnographical fieldwork to gain knowledge and legitimacy, we spent one year codesigning the first version of the game with a group of volunteer villagers. The game, called Kulayijana (teaching each other), was tested and validated by other members of the rural communities and subsequently presented to protected area managers. We show how this approach allowed the negotiation of uncertainties and their inclusion in a model that constitutes a shared representation of farmers' interactions with the protected area. We emphasize the fact that working with marginalized actors first increased participation, appropriation, and confidence of rural communities to engage in a multistakeholder debate, thus reducing power imbalance among actors. We conclude by discussing the next phase of our work: the necessary involvement of conservation actors in the Kulayijana team, and the implementation of Kulayijana with higher hierarchical levels. (Resume d'auteur)

Dynamics of Social Interaction: Kinematic Analysis of a Joint Action.

Non-verbal social interaction between humans requires accurate understanding of the others’ actions. The cognitivist approach suggests that successful interaction depends on the creation of a shared representation of the task, where the pairing of perceptive and motor systems of partners allows inclusion of the other’s goal into the overarching representation. Activity of the Mirror Neurons System (MNS) is thought to be a crucial mechanism linking two individuals during a joint action through action observation. The construction of a shared representation of an interaction (i.e., joint action) depends upon sensorimotor cognitive processes that modulate the ability to adapt in time and space. We attempted to detect individuals’ behavioral/kinematic change resulting in a global amelioration of performance for both subjects when a common representation of the action is built using a repetitive joint action. We asked pairs of subjects to carry out a simple task where one puts a base in the middle of a table and the other places a parallelepiped fitting into the base, the crucial manipulation being that participants switched roles during the experiment. We aimed to show that a full comprehension of a joint action is not an automatic process. We found that, before switching the interactional role, the participant initially placing the base orientated it in a way that led to an uncomfortable action for participants placing the parallelepiped. However, after switching roles, the action’s kinematics by the participant who places the base changed in order to facilitate the action of the other. More precisely, our data shows significant modulation of the base angle in order to ease the completion of the joint action, highlighting the fact that a shared knowledge of the complete action facilitates the generation of a common representation. This evidence suggests the ability to establish an efficient shared representation of a joint action benefits from physically taking our partner’s perspective because simply observing the actions of others may not be enough.

Shared orthographic neuronal representations for spelling and reading

A central question in the study of the neural basis of written language is whether reading and spelling utilize shared orthographic representations. While recent studies employing fMRI to test this question report that the left inferior frontal gyrus (IFG) and ventral occipitotemporal cortex (vOTC) are active during both spelling and reading in the same subjects (Purcell et al., 2011a; Rapp and Lipka, 2011), the spatial resolution of fMRI limits the interpretation of these findings. Specifically, it is unknown if the neurons which encode orthography for reading are also involved in spelling of the same words. Here we address this question by employing an event-related functional magnetic resonance imaging-adaptation (fMRI-A) paradigm designed to examine shared orthographic representations across spelling and reading. First, we identified areas that independently showed adaptation to reading, and adaptation to spelling. Then we identified spatial convergence for these two separate maps via a conjunction analysis. Consistent with previous studies (Purcell et al., 2011a; Rapp and Lipka, 2011), this analysis revealed the left dorsal IFG, vOTC and supplementary motor area. To further validate these observations, we then interrogated these regions using an across-task adaptation technique, and found adaptation across reading and spelling in the left dorsal IFG (BA 44/9). Our final analysis focused specifically on the Visual Word Form Area (VWFA) in the vOTC, whose variability in location among subjects requires the use of subject-specific identification mechanisms (Glezer and Riesenhuber, 2013). Using a functional localizer for reading, we defined the VWFA in each subject, and found adaptation effects for both within the spelling and reading conditions, respectively, as well as across spelling and reading. Because none of these effects were observed during a phonological/semantic control condition, we conclude that the left dorsal IFG and VWFA are involved in accessing the same orthography-specific representations for spelling and reading.

Learning discriminative representations for semantical crossmodal retrieval

Heterogeneous gap among different modalities emerges as one of the critical issues in multimedia retrieval areas. Unlike traditional unimodal cases, where raw features are extracted and directly measured, the heterogeneous nature of crossmodal tasks requires the intrinsic semantic representation to be compared in a unified framework. Based on a flexible “feature up-lifting and down projecting” mechanism, this paper studies the learning of crossmodal semantic features that can be retrieved across different modalities. Two effective methods are proposed to mine semantic correlations. One is for traditional handcrafted features, and the other is based on deep neural network. We treat them respectively as normal and deep version of our proposed shared discriminative semantic representation learning (SDSRL) framework. We evaluate both of these two methods on two public multimodal datasets for crossmodal and unimodal retrieval tasks. The experimental results demonstrate that our proposed methods outperform the compared baselines and achieve state-of-the-art performance in most scenarios.

Processing of irregular polysemes in sentence reading.

The degree to which meanings are related in memory affects ambiguous word processing. We examined irregular polysemes, which have related senses based on similar or shared features rather than a relational rule, like regular polysemy. We tested to what degree the related meanings of irregular polysemes (wire) are represented with shared semantic information versus unshared information represented separately, like homonyms (bank). Monitoring eye fixations, we found that later context supporting the less frequent meaning of an irregular polyseme did not slow down reading compared with control conditions, whereas for homonyms it did. This indicates that in the absence of preceding biasing context, readers access a shared component of an irregular polyseme's representation. Additionally, when the same context words preceded the ambiguous word, both irregular polysemes and homonyms initially elicited longer reading times, but the observed reading slow-down was weaker and less persistent for irregular polysemes than homonyms, indicating less competition between meaning components. We interpret these results as evidence of a shared features representation for irregular polysemes, which additionally incorporates unshared portions of meaning that can compete. When preceding, biasing context is available, readers activate shared and unshared components of the senses, producing a more fully instantiated meaning. (PsycINFO Database Record

Shared Representations Underlie Metaphonological Judgments and Speech Motor Control

Researchers often use metalinguistic judgments to investigate phonological representations. The representations are assumed to govern speech motor control and thereby shape articulatory and acoustic characteristics of speech. Yet little is known about the relationship between metalinguistic judgments, phonological representations, and motor control. This paper reports on an experiment that directly investigates the relation between metalinguistic judgments and articulatory control, hypothesizing that the two share a common representation. This hypothesis predicts that differences in judgments should be correlated with differences in the acoustic characteristics of responses. An experiment was conducted in which syllable count judgments and productions of words with tense vowel/diphthong nuclei and liquid codas were obtained from native speakers of English. A subset of these words have previously been shown to exhibit ­variation in syllable count judgments. Acoustic analyses of productions showed that rime ­durations and formant trajectories differed between words associated with monosyllabic vs. disyllabic syllable count judgments. These results support the hypothesis that a common representation is utilized by the processes responsible for metaphonological judgments of syllable count and speech motor control.

Neural Population Decoding Reveals the Intrinsic Positivity of the Self.

People are motivated to hold favorable views of themselves, which manifests as a positivity bias when evaluating their own performance and abilities. However, it remains an open question whether positive affect is an essential component of people's self-concept. Prior functional neuroimaging research demonstrated that similar regions of the brain support positive affect and self-referential processing, although a direct test of their shared representation has yet to be examined. Here we use functional magnetic resonance imaging in conjunction with multivariate pattern analysis in a cross-domain neural population decoding paradigm. We found that a multivariate pattern classifier model trained to dissociate neural responses to viewing positively and negatively valenced images can dissociate thinking about oneself from a close friend during a lexical trait-judgment task commonly used in the study of self-referential processing. Cross-domain classification accuracy was found to be highest in the ventral medial prefrontal cortex (vMPFC), a region previously implicated in both self-referential processing and positive affect. These results show that brain responses during self-referential processing can be decoded from multi-voxel activation patterns in the vMPFC when viewing positively valenced material, thereby providing evidence that positive affect may be a central component of the mental representation of the self.

Enhancing Global Service-Learning with Partnerships as an Engagement Strategy for Christian Higher Education

ABSTRACTGlobal engagement programming across higher education continues to expand as institutional leaders and practitioners strive to meet global citizenship and civic engagement outcomes. This article presents case study research on a global service-learning partnership, the “Christian University” (CU) Wheelchair Project, which has involved students in a three-semester course sequence that includes research under faculty guidance at a Kenyan school serving 300 children with disabilities. The coauthors participated with the CU students to address the following research questions: (a) What are the characteristics of a Christian Global Service-Learning project that involved partnerships? and (b) What are the observed expectations and impacts of this partnership in the local setting? The case study offers an example of a global service-learning partnership that is grounded in institutional and community input throughout the project's conception and implementation, as well as during assessment of the program. Analysis of the data revealed three components of this program as an effective model of a global service-learning partnership: (a) shared representation throughout the program, (b) valuing both student and community outcomes, and (c) additional perceived community benefits. Corbett and Fikkert's (2012) relief-rehabilitation-development and Fraser's (2009) parity of participation frameworks were used to illustrate why it is necessary to intentionally develop partnerships where the local community can genuinely participate. Drawing from the related literature, the authors argue that global service-learning with partnership programs have the potential to move Christian higher education beyond simply service to transformative and sustained community collaboration in which social justice can be effectively enacted.

I am going to make the most out of it! Italian university Credit Mobility Students' social representations of alcohol use during study abroad experiences.

The aim was to explore shared representations of alcohol use in students who were to travel abroad to study. Focus group data from Italian students ( N = 69) were collected. Analyses used Grounded Theory Methodology and were informed by the four key components of Social Representation Theory (cognition, emotion, attitude and behavioural intentions). The study abroad experience was described as one that would involve an increase in alcohol consumption compared to pre-departure levels. Reasons given included greater social and leisure opportunities involving alcohol, reduced social control and features of the host country environment. Opportunities to intervene and address risky alcohol use in this group are discussed.

Eye fixation during multiple object attention is based on a representation of discrete spatial foci.

We often look at and attend to several objects at once. How the brain determines where to point our eyes when we do this is poorly understood. Here we devised a novel paradigm to discriminate between different models of spatial selection guiding fixation. In contrast to standard static attentional tasks where the eye remains fixed at a predefined location, observers selected their own preferred fixation position while they tracked static targets that were arranged in specific geometric configurations and which changed identity over time. Fixations were best predicted by a representation of discrete spatial foci, not a polygonal grouping, simple 2-foci division of attention or a circular spotlight. Moreover, attentional performance was incompatible with serial selection. Together with previous studies, our findings are compatible with a view that attentional selection and fixation rely on shared spatial representations and suggest a more nuanced definition of overt vs. covert attention.

A shared numerical representation for action and perception.

Humans and other species have perceptual mechanisms dedicated to estimating approximate quantity: a sense of number. Here we show a clear interaction between self-produced actions and the perceived numerosity of subsequent visual stimuli. A short period of rapid finger-tapping (without sensory feedback) caused subjects to underestimate the number of visual stimuli presented near the tapping region; and a period of slow tapping caused overestimation. The distortions occurred both for stimuli presented sequentially (series of flashes) and simultaneously (clouds of dots); both for magnitude estimation and forced-choice comparison. The adaptation was spatially selective, primarily in external, real-world coordinates. Our results sit well with studies reporting links between perception and action, showing that vision and action share mechanisms that encode numbers: a generalized number sense, which estimates the number of self-generated as well as external events.

Visual Space Constructed by Saccade Motor Maps.

How visual space is represented in the brain is an open question in neuroscience. Embodiment theories propose that spatial perception is structured by neural motor maps. Especially, maps which code the targets for saccadic eye movements contain a precise representation of external space. In this review article, we examine how modifications in saccade maps are accompanied by changes in visual space perception. Saccade adaptation, a method which systematically modifies saccade amplitudes, alters the localization of visual objects in space. We illustrate how information about saccade amplitudes is transferred from the cerebellum (CB) to the frontal eye field (FEF). We argue that changes in visual localization after adaptation of saccade maps provide evidence for a shared representation of visual and motor space.

Empathy-Related Responding in Chinese Toddlers: Factorial Structure and Cognitive Contributors.

The critical role of the second year of life in the development of empathy is well accepted by psychologists. However, the developmental trends of the different components of empathy and the potential factors underlying these components during this critical period remain unclear. Eighty‐four Chinese toddlers in the second year of life participated in the present study. Empathy‐related responses were observed during three simulated procedures performed by each child's primary caregiver, the experimenter and a baby doll. An exploratory factor analysis was used to identify the different components of empathy. The shared representation of the self and others was measured using the Tasks for the Observation of Self‐Concept, self/other awareness was measured with a series of mirror tests and inhibitory control was measured using the Cylinder Inhibitory Control Task. The results showed that the empathy of Chinese toddlers contains three factors: sympathy, personal distress and orientation. Potential cognitive factors contribute to the different components of empathy through both independent and joint effects. © 2016 The Authors Infant and Child Development Published by John Wiley & Sons, Ltd.