Bridging the Semantic Sensor Web and Multimodal Human-Machine Interaction Using SCXML

Abstract

This paper presents a novel approach to combine, process and enrich data from heterogeneous sensor networks, and, further, to semantically annotate the observations of the sensed world thus obtained and publish them on the future Semantic Sensor Web. The processing and systematic annotation of sensor data of diverse nature obtained from an ever growing number of different sources that are beginning to shape the Sensor Web is a challenging endeavour. Nevertheless, a similar challenge is inherent to the activity of Human-Machine Interface designers. In this field the SCXML language, based on the concept of Harel statecharts, is being proposed by the W3C as a central element in the design of multimodal interaction managers, which have to deal with information in different communication modalities, integrating and interpreting the user’s input, and coordinating the output among them. We propose extending the use of SCXML to deal also with sensor information, as this language has the capability to combine and process this information as it does with interaction information. SCXML can also coordinate external tools (based on RDF, OGC-SWE’s O&M and OWL) to generate semantic annotations of the knowledge obtained, shareable across applications. This approach can pave the way to develop rich context-aware applications. We illustrate the discussion with a smart-car scenario that we have developed, and present the main implementation details. Keywords: Sensor web, semantic sensor web, human-machine interaction, SCXML, sensor processing, information enrichment, semantic annotation, RDF, Sensor and Actuator Networks (SANs), Internet-of-Things (IoT)