Multimodal Behavior Research Articles

Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions.

The development of large-scale corpora has led to a quantum leap in our understanding of speech in recent years. By contrast, the analysis of massive datasets has so far had a limited impact on the study of gesture and other visual communicative behaviors. We utilized the UCLA-Red Hen Lab multi-billion-word repository of video recordings, all of them showing communicative behavior that was not elicited in a lab, to quantify speech-gesture co-occurrence frequency for a subset of linguistic expressions in American English. First, we objectively establish a systematic relationship in the high degree of co-occurrence between gesture and speech in our subset of expressions, which consists of temporal phrases. Second, we show that there is a systematic alignment between the informativity of co-speech gestures and that of the verbal expressions with which they co-occur. By exposing deep, systematic relations between the modalities of gesture and speech, our results pave the way for the data-driven integration of multimodal behavior into our understanding of human communication.

Assessing intercity multimodal choice behavior in a Touristy City: A factor analysis

This paper investigates influential factors in passengers' intercity multimodal choice behaviors in a touristy city. By collecting large individual-level data through a comprehensive field survey that was carried out at the major transportation hubs in Xi'an, China, we studied four travel modes of the surveyed travelers in this touristy city, including air, high-speed rail, traditional passenger train, and express bus. For factor analysis, 12 variables, including not only individual-related attributes but also ticketing methods and mental perceptions, were used as the independent factors after the correlation analysis and collinearity test. The regression relationships between the travel mode choice and the independent variables were studied using Bayesian multinomial logistic regression. The results indicate that those 12 factors have significant and various influences on passengers' mode choices. In particular, travel distance, fare rate, intercity travel time per hundred kilometers, quality of service, accessibility of transportation hubs, and ticketing methods have influential contributions for explaining the choice decision-makings. The findings demystify the effects of several unexplored factors in intercity multimodal travel choice behaviors and shed new light on formulating traffic management strategies for service providers and decision-makers in practice.

A Machine-Learning-Based Approach to Predict the Health Impacts of Commuting in Large Cities: Case Study of London

The daily commute represents a source of chronic stress that is positively correlated with physiological consequences, including increased blood pressure, heart rate, fatigue, and other negative mental and physical health effects. The purpose of this research is to investigate and predict the physiological effects of commuting in Greater London on the human body based on machine-learning approaches. For each participant, the data were collected for five consecutive working days, before and after the commute, using non-invasive wearable biosensor technology. Multimodal behaviour, analysis and synthesis are the subjects of major efforts in computing field to realise the successful human–human and human–agent interactions, especially for developing future intuitive technologies. Current analysis approaches still focus on individuals, while we are considering methodologies addressing groups as a whole. This research paper employs a pool of machine-learning approaches to predict and analyse the effect of commuting objectively. Comprehensive experimentation has been carried out to choose the best algorithmic structure that suit the problem in question. The results from this study suggest that whether the commuting period was short or long, all objective bio-signals (heat rate and blood pressure) were higher post-commute than pre-commute. In addition, the results match both the subjective evaluation obtained from the Positive and Negative Affect Schedule and the proposed objective evaluation of this study in relation to the correlation between the effect of commuting on bio-signals. Our findings provide further support for shorter commutes and using the healthier or active modes of transportation.

On the multi-mode behavior of vibrating rods attached to nonlinear springs

This paper investigates the harmonic response of vibrating rods with an array of nonlinear springs. The proposed analysis is multi-mode in the sense that the response functions are plotted over wide frequency bands where several resonances can be observed. Particularly, this study aims at investigating the way the vibration modes interact with each other, given the occurrence of local nonlinearities. Also, it aims at investigating the effect of periodic local nonlinearities on the dynamic behavior of the rods, which is closely related to the topic of nonlinear metamaterials. Two approaches are proposed, namely the polynomial method and the perturbation method. The polynomial method uses closed-form solutions of the equation of motion of a rod attached to a small number of springs. This yields a scalar polynomial equation which is well suited for accurately computing the receptance functions at some point of the rod. On the other hand, the proposed perturbation method invokes a subspace projection, which consists in expanding the displacement of the rod on a reduced (finite) basis of vibration modes. This yields a cubic matrix equation which can be easily solved using appropriate solvers. Numerical experiments are carried out which highlight the relevance of both approaches. It is found that the resonance peaks of the rod, once coupled to the nonlinear springs, shift to the high frequencies. This appears to be an interesting feature for the passive control of these systems in the low-frequency range where the vibration levels can be strongly reduced, i.e., compared to the case where purely linear springs are only considered.

Multimode single-pass spatio-temporal squeezing.

We present a single-pass source of broadband multimode squeezed light with potential application in quantum information and quantum metrology. The source is based on a type I parametric down-conversion (PDC) process inside a bulk nonlinear crystal in a non-collinear configuration. The generated squeezed light exhibits a spatio-temporal multimode behavior that is probed using a homodyne measurement with a local oscillator shaped both spatially and temporally. Finally we follow a covariance matrix based approach to reveal the distribution of the squeezing among several independent temporal and spatial modes. This unambiguously validates the multimode feature of our source.

Multivariate count data models for adoption of new transport modes in an organization-based context

This study investigates the potential market demand of shared-ride taxi and shuttle services designed to serve members of organizations in dense urbanized areas. It develops and compares two different multivariate count data modeling approaches, the multinomial distribution and the full enumeration of count alternatives, under an integrated choice and latent variable framework. The study accounts for day-to-day variability in commuting behavior, also known as multimodality, by modeling the weekly frequency of commuting by different travel modes instead of modeling choices for a single trip/day. Using stated preference data collected in the Spring of Academic Year 2016–2017, the models are applied to a case study of students who are highly dependent on private cars at the American University of Beirut (AUB), Lebanon. Policy analysis is conducted to investigate the impact of different price levels and modal attributes on the students’ mode choice behavior. Under practical scenarios, results show that more than 55% of students would adopt a multimodal travel behavior in a given week and that 9–20% of trips are expected to be made by shared-taxi and 12–25% by shuttle. Thus, modeling single trip/day choices instead of weekly decisions would lead to limitations in model forecasts related to the full impact of the proposed policies over longer periods. Results also show that the full enumeration model guarantees higher prediction accuracy and results in an estimate of value of time that is closer to other local estimates for the study area.

Young social milieus and multimodality: interrelations of travel behaviours and psychographic characteristics

ABSTRACT In many countries of the global North, since the 2010s, there have been discussions about young adults turning away from the tendency exclusively to use cars and moving towards more multimodal behaviours, i.e. the flexible use of several transport modes. A more differentiated perspective on the young generation based on the sociostructural criteria of social milieus is presented in this paper with an empirically founded dataset from the city of Dortmund (Germany). Social milieus aim to reflect both horizontal and vertical differentiations and inequalities within society. Drawing on this assumption, multimodal travel behaviours and corresponding psychological assessments of transport modes are analysed with regard to three contrasting young social milieus: i. precariat, ii. middle class, and iii. cosmopolitan milieu. Based on our observations of the three social milieus, the prevailing conceptualisation of young adults as a supposedly ‘homogeneous group’ of key drivers towards a multimodal society must be negated: i) The precariat is exposed to socioeconomic restrictions and limited in its free mode choice. ii) The middle class demonstrates signs of a (conservative) reproduction of car-oriented behaviour patterns. iii) Only the cosmopolitan milieu indicates a less emotional attachment to the private car and favours ‘green’ multimodal behaviours instead.

DeepGoal: Learning to drive with driving intention from human control demonstration

Recent research on automotive driving has developed an efficient end-to-end learning mode that directly maps visual input to control commands. However, it models distinct driving variations in a single network, which increases learning complexity and is less adaptive for modular integration. In this paper, we re-investigate human’s driving style and propose to learn an intermediate driving intention region to relax the difficulties in end-to-end approach. The intention region follows both road structure in image and direction towards goal in public route planner, which addresses visual variations only and figures out where to go without conventional precise localization. Then the learned visual intention is projected on vehicle local coordinate and fused with reliable obstacle perception to render a navigation score map that is widely used for motion planning. The core of the proposed system is a weakly-supervised cGAN-LSTM model trained to learn driving intention from human demonstration. The adversarial loss learns from limited demonstration data with one local planned route and enables reasoning of multi-modal behaviors with diverse routes while testing. Comprehensive experiments are conducted with real-world datasets. Results indicate the proposed paradigm can produce more consistent motion commands with human demonstration and shows better reliability and robustness to environment change. Our code is available at https://github.com/HuifangZJU/visual-navigation.

Modeling I/O performance variability in high-performance computing systems using mixture distributions

Performance variability is an important factor of high-performance computing (HPC) systems. HPC performance variability is often complex because its sources interact and are distributed throughout the system stack. For example, the performance variability of I/O throughput can be affected by factors such as CPU frequency, the number of I/O threads, file size, and record size. In this paper, we focus on the I/O throughput variability across multiple executions of a benchmark program. For a given system configuration, the distribution of throughputs from run to run is of interest. We conduct large-scale experiments and collect a massive amount of data to study the distribution of I/O throughput under tens of thousands of system configurations. Despite normality often being assumed in the literature, our statistical analysis reveals that the performance variability is not normally distributed under most system configurations. Instead, multimodal distributions are common for many system configurations. We propose the use of mixture distributions to describe the multimodal behavior. Various underlying parametric distributions such as normal, gamma, and the Weibull are considered. We apply an expectation–maximization (EM) algorithm for parameter estimation and use the Bayesian information criterion (BIC) for parametric model selections. We also illustrate how to use the estimated mixture distribution to calculate the number of runs needed for future experiments on variability analysis. The paper provides a useful tool set in studying the behavior of performance variability.

The Multimodal Representational Characteristics of Mandarin Children’s Disagreement —— A Case Study Based on Diachronic Videos

Based on a case study, this research collects naturally happened videos of one mandarin-speaking child during the period of her 1.5-3.5 years old, and makes a thorough study of the multimodal representational characteristics of disagreement of this child from the perspective of Multimodal Discourse Analysis. Within a data corpus constituted by 80 hours’ video-recorded life traces of a pre-school child, a corpus of 114 cases of disagreement was selected for a qualitative analysis through the <i>Elan</i> software. It is shown that mandarin-speaking children of this age period usually conduct multimodal communicative behaviors when they are expressing disagreement, the modal forms of which include verbal disagreement, nonverbal disagreement, verbal-nonverbal disagreement. To be more specific, verbal-nonverbal disagreement involves two kinds of modal interactional relationships: equivalent and complementary with the latter containing reinforced and non-reinforced cases. Through this research, parents and early educational staffs are suggested to pay more attention on children’s multimodal expressions on the one hand and to employ more non-verbal resources in interacting with young children on the other hand. What’s more, the point of individual differences of children is a time-consuming but simultaneously a noteworthy one for both parents and preschool teachers. The results of this study bring further light on the knowledge of early education and parental involvement.

Multi-Modal Pedestrian Trajectory Prediction for Edge Agents Based on Spatial-Temporal Graph

Edge agents, represented by socially-aware robots and autonomous vehicles, have gradually been integrated into human society. The safety navigation system in interactive scenes is of great importance to them. The key of this system is that the edge agent has the ability to predict the pedestrian trajectory in the dynamic scene, so as to avoid collision. However, predicting pedestrian trajectories in dynamic scenes is not an easy task, because it is necessary to comprehensively consider the spatial-temporal structure of human-environment interaction, visual attention, and the multi-modal behavior of human walking. In this paper, a scalable spatial-temporal graph generation adversarial network architecture (STG-GAN) is introduced, which can comprehensively consider the influence of human-environment interaction and generate a reasonable multi-modal prediction trajectory. First, we use LSTM nodes to flexibly transform the spatial-temporal graph of human-environment interactions into feed-forward differentiable feature coding, and innovatively propose the global node to integrate scene context information. Then, we capture the relative importance of global interactions on pedestrian trajectories through scaled dot product attention, and use recurrent sequence modeling and generative adversarial network architecture for common training, so as to generate reasonable pedestrian future trajectory distributions based on rich mixed features. Experiments on public data sets show that STG-GAN is superior to previous work in terms of accuracy, reasoning speed and rationality of trajectory prediction.

Analysis of multi-modal commute behavior with feeding and competing ridesplitting services

Public transit is an essential travel mode in many urban areas. Emerging dynamic ridesplitting programs provided by transportation network companies (TNCs) can be a double-edged sword to public transit. On the one hand, the program provides convenient services to solve first- and last-mile problems. On the other hand, long-distance ridesplitting services may also draw passengers away from public transit. In this paper, we propose a network model to analyze multi-modal commute behavior with ridesplitting programs as both feeders and competitors to public transit, which is with limited accessibility to passengers. The ridesplitting priority and ridesplitting fare ratio (i.e., ridesplitting fare over non-ridesplitting fare) are incorporated as operational strategies of the TNC. Through numerical studies, we find that a significant number of public transit passengers will shift to long-distance ridesplitting services under low fare ratios; and a high ridesplitting priority can lead to a demand drawback for long-distance ridesplitting, which raises public transit ridership. To maintain public transit ridership, the TNC needs to keep a high fare ratio and a high priority; meanwhile, the number of short-distance ridesplitting orders can also decrease dramatically, which may lead to a loss in unit time revenue of the TNC. We note that a win–win condition can be reached through a separated discount strategy for first- and last-mile ridesplitting services. Such a strategy can both increase the number of short-distance ridesplitting orders for the TNC and boost transit ridership for the government, as well as provide low-cost services to passengers.

Multi-mode distortion behavior of aluminum alloy thin sheets in immersion quenching

Aluminum alloy thin sheets are widely used to produce light-weight high-strength components and the sheets are generally quenched before forming (Q-F) to improve the final mechanical properties of components. However, the distortion in quenching will significantly affect the quality and stability of the following forming process. In this study, the distortion behavior of 2219 aluminum alloy thin sheets and its forming mechanism in quenching process were investigated through experiments and finite element analysis (FEA). The results indicate that, the quenched thin sheets basically present three distortion modes, namely saddle-shape, shovel-shape and arch-shape. The distortion modes of the quenched thin sheets are determined by the bending modes of distortion zone which is the zone of the thin sheet near the water surface. When the bending mode is one-half sine wave, the quenching thin sheets always show saddle-shape. However, for the cases where the bending mode changes to three-half sine wave and the bending of distortion zone is suppressed, the quenching thin sheets show shovel-shaped and arch-shaped distortion modes, respectively. The reasons behind the variations of the bending mode are finally analyzed based on a buckling criterion under laterally constrained conditions. These results will provide an in-depth understanding of quenching distortion behavior and lay a basic guidance for controlling the distortion of thin sheets in quenching process.

Multimodal travel behaviour, walkability indices, and social mobility: how neighbourhood walkability, income and household characteristics guide walking, biking & transit decisions

ABSTRACTRecent research in city planning has measured “walkability” and multi-modalism through quantification of the built environment. Results have shown correlations with increased property values and neighbourhood stabilisation as well as a lack of spatial distribution of accessible resources based on socio-economic class. Some studies suggest that improvements in these measures can be made, particularly with regard to commute mode diversity. This study tests and scrutinises the association between a walkability index and multimodal travel behaviour – controlling for both built environment and socio-demographic attributes of agents in investigating stated non-vehicular transportation modes. It finds that walkability indexes are connected to multimodal behaviours broadly; whether it be those who walking, bicycling, and or taking transit. It also finds that distance is a significantly associated with transit behaviour in walkable neighbourhoods, but the probability of walking or cycling is tied more to the built environment factors, age, household income and size. These results suggest that cities might benefit from policy efforts that not only continue to address issues of spatial equity but that they might consider policies that promote income mobility to encourage sustainable travel.

Spatial and frequency multimode in the dressing parametric amplified multiwave mixing process

The quantum multimode of correlated fields is essential for future quantum-correlated imaging. Here we investigate multimode properties theoretically and experimentally for the parametric amplified multiwave mixing process. The multimode behavior of the signals in our system stems from spatial phase mismatching caused by frequency resonant linewidth. In the spatial domain, we observe the emission rings with an uneven distribution of photon intensity in the parametric amplified four-wave mixing process, suggesting different spatial modes. The symmetrical distribution of spatial spots indicates the spatial correlation between the Stokes and anti-Stokes signals. While in the frequency domain, the multimode character is reflected as multiple peaks splitting in the signals’ spectrum. A novelty in our experiment, the number of multimodes both in the spatial and frequency domains can be controlled by dressing lasers by modifying the nonlinear susceptibility. Finally, we extend the multimode properties to the multiwave mixing process. The results can be applied in quantum imaging.

Nonstationary Discrete Convolution Kernel for Multimodal Process Monitoring.

Data-driven process monitoring has benefited from the development and application of kernel transformations, especially when various types of nonlinearity exist in the data. However, when dealing with the multimodality behavior that is frequently observed in the process operations, the most widely used radial basis function (RBF) kernel has limitations in describing process data collected from multiple normal operating modes. In this article, we highlight this limitation via a synthesized example. In order to account for the multimodality behavior and improve the fault detection performance accordingly, we propose a novel nonstationary discrete convolution kernel, which derives from the convolution kernel structure, as an alternative to the RBF kernel. By assuming the training samples to be the support of the discrete convolution, this new kernel can properly address these training samples from different operating modes with diverse properties and, therefore, can improve the data description and fault detection performance. Its performance is compared with RBF kernels under a standard kernel principal component analysis framework and with other methods proposed for multimode process monitoring via numerical examples. Moreover, a benchmark data set collected from a pilot-scale multiphase flow facility is used to demonstrate the advantages of the new kernel when applied to an experimental data set.

Neural state machine for character-scene interactions

We propose Neural State Machine , a novel data-driven framework to guide characters to achieve goal-driven actions with precise scene interactions. Even a seemingly simple task such as sitting on a chair is notoriously hard to model with supervised learning. This difficulty is because such a task involves complex planning with periodic and non-periodic motions reacting to the scene geometry to precisely position and orient the character. Our proposed deep auto-regressive framework enables modeling of multi-modal scene interaction behaviors purely from data. Given high-level instructions such as the goal location and the action to be launched there, our system computes a series of movements and transitions to reach the goal in the desired state. To allow characters to adapt to a wide range of geometry such as different shapes of furniture and obstacles, we incorporate an efficient data augmentation scheme to randomly switch the 3D geometry while maintaining the context of the original motion. To increase the precision to reach the goal during runtime, we introduce a control scheme that combines egocentric inference and goal-centric inference. We demonstrate the versatility of our model with various scene interaction tasks such as sitting on a chair, avoiding obstacles, opening and entering through a door, and picking and carrying objects generated in real-time just from a single model.

K-space analysis of complex large-scale meta-structures using the Inhomogeneous Wave Correlation method

Abstract An inverse wavenumber identification tool is used to characterize the vibration behavior of three structures and meta-structures with different complexity levels: a plane steel panel, a curved thick composite sandwich shell and a stiffened aluminum aircraft sidewall panel. Bare structures are first studied and then equipped with spatially distributed small-scale resonators, leading to meta-structures. For the two curved panels, tests are conducted under diffuse acoustic field and point mechanical excitations. For each studied case, the effect of the industrially-oriented small-scale resonators is highlighted using frequency and wavenumber analysis, showing general attenuation of the vibration level and even band gaps occurrence. The complex wavenumber identification allows also estimating the structural loss factor in the composite sandwich panel, while the multi-modal behavior is captured in the aluminum aircraft sidewall panel.

Computational Study of Primitive Emotional Contagion in Dyadic Interactions

Interpersonal human-human interaction is a dynamical exchange and coordination of social signals, feelings and emotions usually performed through and across multiple modalities such as facial expressions, gestures, and language. Developing machines able to engage humans in rich and natural interpersonal interactions requires capturing such dynamics. This paper addresses primitive emotional contagion during dyadic interactions in which roles are prefixed. Primitive emotional contagion was defined as the tendency people have to automatically mimic and synchronize their multimodal behavior during interactions and, consequently, to emotionally converge. To capture emotional contagion, a cross-recurrence based methodology that explicitly integrates short and long-term temporal dynamics through the analysis of both facial expressions and sentiment was developed. This approach is employed to assess emotional contagion at unimodal, multimodal and cross-modal levels and is evaluated on the Solid SAL-SEMAINE corpus. Interestingly, the approach is able to show the importance of the adoption of cross-modal strategies for addressing emotional contagion.

Editorial

The origin of this special issue took place at the 9th European Nonlinear Dynamics Conference (ENOC 2017) in Budapest, Hungary. Specifically, the mini-symposium on Random Dynamical Systems – Recent Advances and New Directions brought together novel perspectives on analyzing stochastic dynamics with applications including biology, structural dynamics, control, energy and mechanics. The expanded use of stochasticity in more realistic models exposes questions related to bifurcations, meta-stability, tipping and early warning signals, multiscale dynamics, and connections between chaos and stochastic dynamics. The observed phenomena in applications drive new methodologies and analyses, needed to understand the interplay between different sources of stochastic effects and nonlinearities, network structure, multi-mode and multi-scale behavior, non-smooth dynamics, energy transfer, and spatio-temporal phenomena. Of course, a single issue cannot hope to cover all of the new topics in stochastic analysis for applications. Nevertheless, we hope that the collection of applications and stochastic models presented in this issue illustrates some of the exciting advances and perspectives relevant for broad classes of stochastic models and demonstrates the need in advancing the theory of stochastic processes.