Multimodal Dialogs Research Articles

English for tourism and AR-assisted context-aware ubiquitous learning: a preliminary design-based research study

With the rise of augmented reality (AR) and context-aware ubiquitous learning (CAUL), pedagogical designers in computer assisted language learning are increasingly developing authentic English for Specific Purposes (ESP) learning environments. However, there has been little research regarding the development of evidence-based principles for English for tourism purposes (ETP) through AR-based CAUL. The purpose of this study was to provide a preliminary analysis of the design principles and develop a learning model based on a rigorous four-phase Design-Based Research procedure. The researchers first developed a location-based AR application for ETP and formulated the first set of design principles. This application was further tested and refined in an iterative process, with data collected from three different sources. The data collection methods included an online questionnaire, onsite observations, and semi-structured expert interviews. All the data was triangulated and analyzed, and a team of experts evaluated the design principles. Results showed that the technical affordances of mobile AR devices should form the basis for the design. The application needs to seamlessly provide multimodal scenery-based AR learning supports to fulfill individual needs in blended learning contexts. Furthermore, a user-friendly interface with personalized functions and a portfolio recording the learning progress needs to be included. Multimodal authentic ETP dialogs and terminology are also necessary in order to build students’ ETP speaking competence. Moreover, the learning material should be compatible with the main theories which drive language learning in CAUL. In light of the findings, a learning model was established to guide AR-based ETP learning and serve as template for future studies in other ESP fields.

Using Vision and Speech Features for Automated Prediction of Performance Metrics in Multimodal Dialogs

Predicting and analyzing multimodal dialog user experience (UX) metrics, such as overall call experience, caller engagement, and latency, among other metrics, in an ongoing manner is important for evaluating such systems. We investigate automated prediction of multiple such metrics collected from crowdsourced interactions with an open-source, cloud-based multimodal dialog system in the educational domain. We extract features from both the audio and video signals and examine the efficacy of multiple machine learning algorithms in predicting these performance metrics. The best performing audio features consist of multiple low-level audio descriptors—intensity, loudness, cepstra, pitch, and so on—and their functionals, extracted using the OpenSMILE toolkit, while the video features are bags of visual words that use 3D Scale-Invariant Feature Transform descriptors. We find that our proposed methods outperform the majority vote classification baseline in predicting various UX metrics rated by both the user and experts. Our results suggest that such automated prediction of performance metrics can not only inform the qualitative and quantitative analysis of dialogs but also be potentially incorporated into dialog management routines for positively impacting UX and other metrics during the course of the interaction.

Recognizing signals of social attitude in interacting with Ambient Conversational Systems

Ambient Intelligence systems require a natural and personalized experience in interacting with services provided by the environment. In this view, the interaction may happen either in a pervasive way, through a combination of devices embedded in the environment, or using a conversational interface acting as an environment concierge. In the latter case, the interface can be embodied in a conversational agent able to involve users in a human-like conversation and to establish a social relation with them. Developing such an Ambient Conversational System (ACS) requires a model of the user that considers not only the cognitive ingredients of his mental state, but also extra-rational factors such as affect, engagement, attitudes. This paper describes a multimodal framework for recognizing the social attitude of users during the interaction with an embodied agent in a smart environment. In particular, we started from the analysis and annotation of advisory dialogs between humans and then we used the annotated corpus to build a framework for recognizing the social attitude in multimodal dialogs with an ACS. Results of the study show an acceptable performance of the framework in recognizing and monitoring the social attitude during the dialog with an ACS. We also compared results of the analysis of human-human interactions with respect to the human-ACS interaction and, even if the level of initiative of subjects during the dialog was lower in this latter modality, the difference in the average number of social moves was not significant, thus showing that subjects probably were keen to establish a social relation with the conversational agent.

The Arrow-Semantics Interpreter

People often sketch diagrams to facilitate their communication. If computers understood such diagrams, people could interact with information systems more intuitively using similar multimodal dialogs that include sketched diagrams. It remains a challenging problem, however, to make computers interpret arrow symbols, which are a frequent and versatile ingredient of such diagrams. This paper develops an algorithm for deducing possible semantic roles of arrow symbols. The Arrow-Semantics Interpreter (ASI) analyzes the elements that surround an arrow symbol and their spatial arrangements. It distinguishes four different semantic roles of arrow symbols—orientation, behavioral description, annotation, and association—each with distinctive requirements on the arrangement of the surrounding elements. The combination of these properties, together with rules for optional elements, yields the potential semantic roles of each arrow symbol from the pattern of its surrounding elements. The assessment shows the ASI derives for each arrow symbol only 1.31 potential semantic roles, which include for 79% of the analyzed diagrams the correct answer. This result indicates that the patterns of elements around arrow symbols are highly useful for deriving the interpretation of arrow symbols.

A scalable formal method for design and automatic checking of user interfaces

The article addresses the formal specification, design and implementation of the behavioral component of graphical user interfaces. The complex sequences of visual events and actions that constitute dialogs are specified by means of modular, communicating grammars called VEG (Visual Event Grammars), which extend traditional BNF grammars to make them more convenient to model dialogs.A VEG specification is independent of the actual layout of the GUI, but it can easily be integrated with various layout design toolkits. Moreover, a VEG specification may be verified with the model checker SPIN, in order to test consistency and correctness, to detect deadlocks and unreachable states, and also to generate test cases for validation purposes.Efficient code is automatically generated by the VEG toolkit, based on compiler technology. Realistic applications have been specified, verified and implemented, like a Notepad-style editor, a graph construction library and a large real application to medical software. It is also argued that VEG can be used to specify and test voice interfaces and multimodal dialogs. The major contribution of our work is blending together a set of features coming from GUI design, compilers, software engineering and formal verification. Even though we do not claim novelty in each of the techniques adopted for VEG, they have been united into a toolkit supporting all GUI design phases, that is, specification, design, verification and validation, linking to applications and coding.