Multimodal Events Research Articles

Ostensive signals support learning from novel attention cues during infancy.

Social attention cues (e.g., head turning, gaze direction) highlight which events young infants should attend to in a busy environment and, recently, have been shown to shape infants' likelihood of learning about objects and events. Although studies have documented which social cues guide attention and learning during early infancy, few have investigated how infants learn to learn from attention cues. Ostensive signals, such as a face addressing the infant, often precede social attention cues. Therefore, it is possible that infants can use ostensive signals to learn from other novel attention cues. In this training study, 8-month-olds were cued to the location of an event by a novel non-social attention cue (i.e., flashing square) that was preceded by an ostensive signal (i.e., a face addressing the infant). At test, infants predicted the appearance of specific multimodal events cued by the flashing squares, which were previously shown to guide attention to but not inform specific predictions about the multimodal events (Wu and Kirkham, 2010). Importantly, during the generalization phase, the attention cue continued to guide learning of these events in the absence of the ostensive signal. Subsequent experiments showed that learning was less successful when the ostensive signal was absent even if an interesting but non-ostensive social stimulus preceded the same cued events.

The associative structure of memory for multi-element events.

The hippocampus is thought to be an associative memory “convergence zone,” binding together the multimodal elements of an experienced event into a single engram. This predicts a degree of dependency between the retrieval of the different elements comprising an event. We present data from a series of studies designed to address this prediction. Participants vividly imagined a series of person–location–object events, and memory for these events was assessed across multiple trials of cued retrieval. Consistent with the prediction, a significant level of dependency was found between the retrieval of different elements from the same event. Furthermore, the level of dependency was sensitive both to retrieval task, with higher dependency during cued recall than cued recognition, and to subjective confidence. We propose a simple model, in which events are stored as multiple pairwise associations between individual event elements, and dependency is captured by a common factor that varies across events. This factor may relate to between-events modulation of the strength of encoding, or to a process of within-event “pattern completion” at retrieval. The model predicts the quantitative pattern of dependency in the data when changes in the level of guessing with retrieval task and confidence are taken into account. Thus, we find direct behavioral support for the idea that memory for complex multimodal events depends on the pairwise associations of their constituent elements and that retrieval of the various elements corresponding to the same event reflects a common factor that varies from event to event.

Novels, nests and other provocations: emergent literacy curriculum production in a childcare centre

In a bid to identify and gain analytic insight into the make-up and dynamics of kindergarten literacy curricula in an era of early childhood education and care reform, this study was designed to trace how classroom literacy curricula were produced in a kindergarten in a childcare centre in Ontario, Canada. Drawing on actor-network theory’s (ANT) definition of curriculum as a network effect and understandings of literacy from multiliteracies theory, this case study using ethnographic tools focused on the ways in which children’s interests and funds of knowledge (i.e. their linguistic, modal, epistemic and cultural resources) were implicated in the production of curriculum in a kindergarten milieu outside of school. Data were collected through observation, interview and document collection methods and were analysed through an ANT approach. Findings identified actors believed to be involved in literacy curriculum production which included an emergent curricular orientation, novels, educators’ understandings of children’s interests and funds of knowledge and materials and life forms from the natural world. The findings also offer a rich illustration of emergent literacy curriculum in action characterized by multimodal classroom literacy events and featuring Read-Aloud.

Investigating the Relation Between Infants' Manual Activity With Objects and Their Perception of Dynamic Events

We examined the relation between 6‐ and 7‐month‐old infants' (N = 60) manual activity with objects during free play and their perception of the features of dynamic, multimodal events. Infants were habituated to a single event in which a hand reached for and manipulated a colorful, multifeatured object, and a sound was heard (e.g., a hand squeezed a purple round object, causing a whistling sound) and then their response to events that involved a change in the appearance of the object, the action, or the sound was assessed. Infants responded least to changes in the appearance of the objects, and their sensitivity to this feature was related to their manual activity with objects during free play. Infants' responding to changes in the sound or action was unrelated to motor activity, suggesting that at this age motor achievements related to object exploration are associated with infants' perception of some, but not all, object features.

Hot Topic Propagation Model and Opinion Leader Identifying Model in Microblog Network

As the network technique is fast developing, the microblog has been a significant carrier representing the social public opinions. Therefore, it is important to investigate the propagation characteristics of the topics and to unearth the opinion leaders in Micro-blog network. The propagation status of the hot topics in the Micro-blog is influenced by the authority of the participating individuals. We build a time-varying model with the variational external field strength to simulate the topic propagation process. This model also fits for the multimodal events. The opinion leaders are important individuals who remarkably influence the topic discussions in its propagation process. They can help to guide the healthy development of public opinion. We build an AHP model based on the influence, the support, and the activity of a node, as well as a microblog-rank algorithm based on the weighted undirected network, to unearth and analyze the opinion leaders’ characteristics. The experiments in the data, collected from the Sina Micro-blog from October 2012 to November 2012 and from January 2013 to February 2013, show that our models predict the trend of hot topic efficiently and the opinion leaders we found are reasonable.

An Empiric Evaluation of a Real-Time Robot Dancing Framework based on Multi-Modal Events

Musical robots have already inspired the creation of worldwide robotic dancing contests, as RoboCup-Junior's Dance, where school teams, formed by children aged eight to eighteen, put their robots in action, performing dance to music in a display that emphasizes creativity of costumes and movement. This paper describes and assesses a framework for robot dancing edutainment applications. The proposed architecture enables the definition of choreographic compositions, which result on a conjunction of reactive dancing motions in real-time response to multi-modal inputs. These inputs are shaped by three rhythmic events (representing soft, medium, and strong musical note-onsets), different dance floor colors, and the awareness of the surrounding obstacles. This layout was applied to a Lego-NXT humanoid robot, built with two Lego-NXT kits, and running on a hand-made dance stage. We report on an empirical evaluation over the overall robot dancing performance made to a group of students after a set of live demonstrations. This evaluation validated the framework's potential application in edutainment and its ability to sustain the interest of the general audience by offering a reasonable compromise between musical-synchrony, animacy and dance performance’s variability. DOI: http://dx.doi.org/10.11591/telkomnika.v10i8.1327 Full Text: PDF

Segregated audio–tactile events destabilize the bimanual coordination of distinct rhythms

We examined to what extent the CNS can efficiently bind together the perception of non-coincident multimodal events with coordinated movements. To do so, we selected a bimanual coordination with left-right asymmetry, which was, achieving 3:2 polyrhythmic movements. We asked participants to synchronize left and right fingers' movements to events presented, respectively, to the left and to the right side. In two segregated conditions, sound was presented on one side at one frequency while touch was presented on the other side at the other frequency; thus, the left and right rhythms were paced via a distinct sensory modality. In the three control conditions, the stimuli on both sides were presented via the same sensory modality: sound, touch, or coincident sound and touch. Our aim was to contrast two opposing hypotheses: Sensory segregated pacing (1) stabilizes polyrhythmic coordination because it favors the distinction between the fast and the slow rhythm versus (2) destabilizes polyrhythmic coordination because it introduces a very strong asymmetry. We performed a parametric study in which the ability to maintain the polyrhythmic coordination was explored over a broad range of pacing rates. We found that switches from the polyrhythmic coordination to an isofrequency pattern took place only in the sensory segregated conditions, at the highest frequencies. Moreover, transitions were preceded by an increase in the variability of the synchronization of movement to stimuli. We therefore propose that the destabilization originating from the asymmetry between sensory modalities overrides the assumed segregation effect. We discuss the possible neuronal underpinnings of this failure of binding of movement to segregated sound and touch.

Interaction of Perceptual Grouping and Crossmodal Temporal Capture in Tactile Apparent-Motion

Previous studies have shown that in tasks requiring participants to report the direction of apparent motion, task-irrelevant mono-beeps can “capture” visual motion perception when the beeps occur temporally close to the visual stimuli. However, the contributions of the relative timing of multimodal events and the event structure, modulating uni- and/or crossmodal perceptual grouping, remain unclear. To examine this question and extend the investigation to the tactile modality, the current experiments presented tactile two-tap apparent-motion streams, with an SOA of 400 ms between successive, left-/right-hand middle-finger taps, accompanied by task-irrelevant, non-spatial auditory stimuli. The streams were shown for 90 seconds, and participants' task was to continuously report the perceived (left- or rightward) direction of tactile motion. In Experiment 1, each tactile stimulus was paired with an auditory beep, though odd-numbered taps were paired with an asynchronous beep, with audiotactile SOAs ranging from −75 ms to 75 ms. Perceived direction of tactile motion varied systematically with audiotactile SOA, indicative of a temporal-capture effect. In Experiment 2, two audiotactile SOAs—one short (75 ms), one long (325 ms)—were compared. The long-SOA condition preserved the crossmodal event structure (so the temporal-capture dynamics should have been similar to that in Experiment 1), but both beeps now occurred temporally close to the taps on one side (even-numbered taps). The two SOAs were found to produce opposite modulations of apparent motion, indicative of an influence of crossmodal grouping. In Experiment 3, only odd-numbered, but not even-numbered, taps were paired with auditory beeps. This abolished the temporal-capture effect and, instead, a dominant percept of apparent motion from the audiotactile side to the tactile-only side was observed independently of the SOA variation. These findings suggest that asymmetric crossmodal grouping leads to an attentional modulation of apparent motion, which inhibits crossmodal temporal-capture effects.

Pathophysiological Roles of Chemokines in Human Reproduction: An Overview

Chemokines are a group of small cytokines that have an ability to induce leukocyte migration. Chemokines exert their functions by binding and activating specific G protein-coupled receptors. Studies have unveiled pleiotropic bioactivities of chemokines in various phenomena ranging from immunomodulation, embryogenesis, and homeostasis to pathogenesis. In the mammalian reproductive system, chemokines unexceptionally serve in multimodal events that are closely associated with establishment, maintenance, and deterioration of fecundity. The aim of this review is to update the knowledge on chemokines in male and female genital organs, with a focus on their potential pathophysiological roles in human reproduction.

No two cues are alike: Depth of learning during infancy is dependent on what orients attention

Human infants develop a variety of attentional mechanisms that allow them to extract relevant information from a cluttered multimodal world. We know that both social and nonsocial cues shift infants’ attention, but not how these cues differentially affect learning of multimodal events. Experiment 1 used social cues to direct 8- and 4-month-olds’ attention to two audiovisual events (i.e., animations of a cat or dog accompanied by particular sounds) while identical distractor events played in another location. Experiment 2 directed 8-month-olds’ attention with colorful flashes to the same events. Experiment 3 measured baseline learning without attention cues both with the familiarization and test trials (no cue condition) and with only the test trials (test control condition). The 8-month-olds exposed to social cues showed specific learning of audiovisual events. The 4-month-olds displayed only general spatial learning from social cues, suggesting that specific learning of audiovisual events from social cues may be a function of experience. Infants cued with the colorful flashes looked indiscriminately to both cued locations during test (similar to the 4-month-olds learning from social cues) despite attending for equal duration to the training trials as the 8-month-olds with the social cues. Results from Experiment 3 indicated that the learning effects in Experiments 1 and 2 resulted from exposure to the different cues and multimodal events. We discuss these findings in terms of the perceptual differences and relevance of the cues.

Development of Multimodal Processing in Infancy.

A number of studies have investigated infants' abilities to extract and discriminate number from multimodal events. These results have been mixed for several possible reasons, including aspects of the experimental design that provide perceptual cues that are unrelated to number, and are known to influence looking preferences. This experiment used a preferential looking paradigm to investigate whether 6- to 9-month-old infants can extract the amodal property of number from an arbitrarily related multimodal event sequence when nonnumerical confounds are removed. Results demonstrate that female infants discriminate number from a multimodal presentation by 6 months, whereas males do so by 8 months. Further, the study underscores the importance of controlling for low-level perceptual cues in looking time experiments aimed at examining infants' cognitive abilities.

Magnetohydrodynamic seismology of solar and stellar coronae

The review presents the study of long period (longer than a few seconds) wave and oscillation phenomena in the corona of the Sun in radio, VL, EUV and X-ray bands, interpreted in terms of MHD theory. This study provides us with the observational foundation for the remote diagnostics of solar and stellar coronae with MHD waves. Kink and sausage magnetoacoustic modes and longitudinal (or acoustic) modes of coronal loops have already been confidently identified in the data. At present, the main research emphasis is put on the determination of parametric relations in the wave and oscillatory phenomena, the study of multi-modal events, and on the identification of MHD modes in quasi-periodic pulsations of solar and stellar flares.

Temporal characteristization of auditory-visual coupling in speech

This work examines the coupling between the acoustic and visual components of speech as it evolves through time. Previous work has shown a consistent correspondence between face motion and spectral acoustics, and between fundamental frequency (F0) and rigid body motion of the head [Yehia et al. (2002), JPHON, 30, 555–568]. Although these correspondences have been estimated both for sentences and for running speech, the analyses have not taken into account the temporal structure of speech. As a result, the role of temporal organization in multi-modal speech cannot be estimated. The current study is a first effort to correct this deficit. We have developed an algorithm, based on recurrent correlation, that computes the correlation between measurement domains (e.g., head motion and F0) as a time-varying function. Using this method regions of high or low correlation, or of rapid transition (e.g., from high to low) can be associated with visual and auditory events. The analysis of the time-varying coupling of multi-modal events has implications for speech planning and synchronization between speaker and listener.

Multimodal Virtual Environments: Response Times, Attention, and Presence

Multimodal virtual environments (VE) succeed better than single-sensory technologies in creating a sense of presence. We hypothesize that the underlying cognitive mechanism is related to a faster mental processing of multimodal events. Comparing simple detection times of unimodal (auditory, visual, and haptic) events, with bimodal and trimodal combinations, we show that mental processing times are in the following order: unimodal > bimodal > trimodal. Given this processing-speed advantage, multimodal VE users start their cognitive process faster, thus, in a similar exposure time they can pay attention to more informative cues and subtle details in the environment and integrate them creatively. This richer, more complete and coherent experience may contribute to an enhanced sense of presence.

Reinforcing robot perception of multi-modal events through repetition and redundancy and repetition and redundancy

For a robot to be capable of development it must be able to explore its environment and learn from its experiences. It must find (or create) opportunities to experience the unfamiliar in ways that reveal properties valid beyond the immediate context. In this paper, we develop a novel method for using the rhythm of everyday actions as a basis for identifying the characteristic appearance and sounds associated with objects, people, and the robot itself. Our approach is to identify and segment groups of signals in individual modalities (sight, hearing, and proprioception) based on their rhythmic variation, then to identify and bind causally-related groups of signals across different modalities. By including proprioception as a modality, this cross-modal binding method applies to the robot itself, and we report a series of experiments in which the robot learns about the characteristics of its own body.

Rejoinder - Bimodal emotion perception: integration across separate modalities, cross-modal perceptual grouping or perception of multimodal events?

The combined perception of affective information provided by seeing a facial expression and hearing a tone of voice presents us with a phenomenon that has received little systematic attention in the laboratory. We argue that although the FLMP appears good at modelling the presently available data, more complex theoretical questions must be raised. We discuss the theoretical shortcomings of the FLMP and envisage two other theoretical frameworks that could guide future research.

Mind model for multimodal communicative creatures and humanoids

This paper presents a computational model of real - time task - oriented dialog skills . The model , termed Ymir, bridges multimodal perception and multimodal action and supports the creation of autonomous computer characters that afford full - duplex , real - time face - to - face interaction with a human . Ymir has been prototyped in software , and a humanoid created , called Gandalf, capable of fluid multimodal dialog . Ymir demonstrates several new ideas in the creation of communicative computer agents , including perceptual integration of multimodal events, distributed planning and decision making, an explicit handling of real time, and perceptuo-motor system layered and motor control with human characteristics. This paper describes the model's architecture and explains its main elements . Examples of implementation and performance are given , and the architecture's limitations and possibilities are discussed .

Sex, mayhem, and psychology’s two solitudes.

Sex, Mayhem, and Psychology's Two Solitudes Review of M. Sabourin, F. Craik, & M. Robert (Eds.), Advances in psychological science, Volume 2: Biological and cognitive aspects. Hove, East Sussex; Psychology Press, 1998. (633 + xxii pages). D. J. K. MEWHORT, Queen's University The volume arose from the International Congress of Psychology held in Montreal, August 1996. It is a collection of papers, predominantly on biological aspects of psychology. Although the title promises a discussion of cognition, like the dog that did not bark in the Sherlock Holmes story, reference to much of current cognitive psychology is conspicuously missing. In the section on Cognition, Perception, and Memory, for example, only three papers (Hatano on comprehension, Bertelson on perception of multimodal events, and Koria on metamemory) have an experimentalbehavioural focus. Moreover, such icons of cognitive psychology as J. R. Anderson, A. Baddeley, C. Burgess, M. Coltheart, D. Hintzman, M. Humphreys, P. Johnson-Laird, H. Pashler, D. Kahneman, W. Kintsch, T. Landauer, B. Murdock, G. McKoon, D. Medin, J. Nairne, R. Nosofsky, A. Newell, R. Ratcliff, R. Schank, R. Shiffrin, A. Treisman, and A. Tversky are not cited anywhere. Connectionist thinking is considered only in Plunkett's paper on cognitive development. In short, the psychological science advanced in the book is a very different psychology than the one I know. As I read the book, I felt like an outsider looking in on a psychology that has been untouched by the revolution in computational modelling and, indeed, untouched by much of current behavioural work. But first, let me turn to sex and mayhem. Roubertoux, Mortaud, Torjman, Le Roy, and Degrelle discuss genetic aspects of aggression using the mouse as an animal model. It is a fascinating example of basic science and, importantly, of good methodology. The genetic contribution to aggression is complex and indirect. As Roubertoux et al. note, it is not a story that lends itself to simple-minded excursions into social policy: It should discourage any eugenic temptation and disgrace eugenics itself' (p. 24). But, as they also note, mayhem results if the lesson is ignored. Their example concerns a dispute within the Behavior Genetics Association leading to the resignation of the President Elect both from office and from the society itself. But, with eugenic and racist views waiting in the wings, even among academics who ought to know the science better, it takes little imagination to project mayhem to a larger social stage. Corballis provides an elegantly written speculation on the mind's evolution. The paper ranges over a wide set of issues from bipedalism, language in nonhuman primates, the development of human language, to specialization of the cerebral hemispheres. It is a delightful romp through anthropology and natural history. But, the speculation is based on indirect correlational evidence. The history of experimental design, from Claude Bernard forward, shows that there is no substitute for manipulative control (a like concern led the Societe Linguistique de Paris in 1866 to ban papers on the origins of language). However delightful and colourful the prose, it is difficult to imagine a practical way to test the idea that the inhabitants of the Great Rift valley adopted bipedal walking because knuckle-walking may have been simply less efficient than upright walking, especially over long distances (p. 39). I raise the methodological issue not because I disliked Corballis's paper but because it was fun and because I felt guilty for liking it: Methodology is the core of science. The topic, the evolution of the mind, is unlikely to engender the emotional response given to sex, race, and criminal aggression, but, as Roubertoux et al. document, once one weakens the rules of evidence, the door is open to mayhem. Kanwisher reviews studies of human brain imaging to illuminate the processing of visual shape, the what-where issue, and face perception. …

Brief report: Preferential looking in intermodal perception by children with autism

Recent reviews of autism have debated whether autism should beconsidered primarily as a disorder of affective and social relations asKanner proposed (see Hobson, 1989) or as a cognitive deficit, specificallyin the ability to construct a "theory of mind" (Baron-Cohen, Leslie, F Leslie, 1987). As Ozonoff, Pennington, and Rogers (1990)have noted, to answer basic questions about autism we must determinewhich features of autism are universal and primary. Consequently, we mustdevelop tasks that can be used irrespective of age or verbal abilities. Thestudy reported here is a first step towards extending methods used withnormal infants to examine the functioning of children with autism. Suchprocedures should be particularly well suited for use with children withautism because the tasks do not require sophisticated responses and aretypically less intrusive.An intermodal preference task was selected for the present study be-cause of its utility in the study of infants' cross-modal abilities and emotionperception, both of which have been implicated in the syndrome of autism(Bryson, 1972; Hobson, Ouston, & Lee, 1988). The intermodal preferencemethod combines the potential for presenting dynamic, multimodal events

TIMING MULTIMODAL EVENTS IN PIGEONS

The peak procedure was used in two experiments to study pigeons' ability to time multimodal events. In the first experiment, birds were trained to time a single event consisting of a 9-s tone or light followed by a 21-s fixed interval associated with a signal of light or tone (signal of the other modality). On occasional empty trials, different lengths of the first signal were followed by a long period of the second signal. Peak response times as a function of the duration of the first signal were linear and had a slope of close to one in all birds. This indicates that the birds were timing only the second signal. In a second experiment, two complex events were used in training. One consisted of a 9-s tone or light followed by a 21-s fixed interval associated with a light or tone. The other consisted of a 21-s tone or light followed by a 9-s fixed interval associated with a light or tone. Different durations of the first signal were again used on empty trials. Peak response times as a function of the duration of the first signal were again linear in all birds. The slope of the function was less than one but greater than zero for 3 birds. This indicates that these birds were partly timing the entire complex event of 30-s duration and partly timing only the second signal of the event. A model is proposed in which the bird takes as a criterion for timing a weighted average of different target criteria. Comparisons with the performance of rats are made.