Interaction In Smart Environments Research Articles

Stream your brain! Speculative economy of the IoT and its pan-kinetic dataveillance

It is now a common belief that the truths of our lives are hidden in the databases streamed from our interactions in smart environments. In this current hype of big data, the Internet of Things has been suggested as the idea to embed small sensors and actuators everywhere to unfold the truths beneath the surfaces of everything. However, remaining the technology that promises more than it can provide thus far, more important for the IoT’s actual expansion to various social domains than the actual discovery of hidden truths has been people’s speculations about the unknown problems, such as hidden security issues or lifestyle concerns, beyond the narrow human knowability but assumed to leave their traces in the IoT-collected big data. This paper discuss this speculation as the concealed cognitive labor of IoT users that projects some fictitious values to the big data IoT companies accumulate. By the term pan-kinetics, the systemic operation of smart actuators is analyzed as the process through which fictitious values of data are converted to the real values as these actuators draw some profitable correlations from physical domains of the IoT. Analyzing smart electroencephalogram (EEG) headsets as the unique IoT devices operating on human brains, it argues how the IoT translates this speculative realism of unknown problems into its big data, which the IoT developers believe to be full of machine-learnable correlations that would lead to the smart solutions of the problems.

Mid-Air Gesture Control of Multiple Home Devices in Spatial Augmented Reality Prototype

Touchless, mid-air gesture-based interactions with remote devices have been investigated as alternative or complementary to interactions based on remote controls and smartphones. Related studies focus on user elicitation of a gesture vocabulary for one or a few home devices and explore recommendations of respective gesture vocabularies without validating them by empirical testing with interactive prototypes. We have developed an interactive prototype based on spatial Augmented Reality (AR) of seven home devices. Each device responds to touchless gestures (identified from a previous elicitation study) via the MS Kinect sensor. Nineteen users participated in a two-phase test (with and without help provided by a virtual assistant) according to a scenario that required from each user to apply 41 gestural commands (19 unique). We report on main usability indicators: task success, task time, errors (false negative/positives), memorability, perceived usability, and user experience. The main conclusion is that mid-air interaction with multiple home devices is feasible, fairly easy to learn and apply, and enjoyable. The contributions of this paper are (a) validation of a previously elicited gesture set; (b) development of a spatial AR prototype for testing of mid-air gestures, and (c) extensive assessment of gestures and evidence in favor of mid-air interaction in smart environments.

SAPIENS

We present SAPIENS, a software architecture designed to support engineering of interactive systems featuring peripheral interaction in the context of smart environments. SAPIENS introduces dedicated components for user and device tracking, attention detection, priority management for devices, tasks, and notifications, context-awareness inference, user interruptibility prediction, and device interchangeability that can be instantiated at will according to the needs of the application. To implement these components effectively, SAPIENS employs event-based processing by reusing the core engine of a recently introduced software architecture, Euphoria (Schipor et al., 2019), that was specifically designed for engineering interactions in smart environments with heterogeneous I/O devices, and relies entirely on web standards, protocols, and open data-interchange formats, such as JavaScript, WebSockets, HTTP, and JSON. This inheritance makes SAPIENS flexible and adaptable to support implementation of diverse application scenarios for peripheral interaction and for a wide variety of smart environments, devices, platforms, data formats, and contexts of use. We present our design criteria for SAPIENS regarding (1) event handling techniques, (2) quality, (3) contextual, and (4) attention-related properties, and describe its components and dataflows that make SAPIENS a specialized software architecture for peripheral interaction scenarios. We also demonstrate SAPIENS with a practical application, inspired and adapted from Bakker's (2013) classical example for peripheral interaction, for which we provide an online simulation tool that researchers and practitioners can readily use to consult actual JavaScript code implementing the inner logic of selected components of our architecture as well as to observe live JSON messages exchanged by the various components of SAPIENS.

Dynamic and Constraint-based building of Ambient Applications with Self-Reflective Smart Objects

Abstract: The increasing availability of novel smart objects in the Internet of Things offers exciting possibilities for a plethora of use cases. Smart environments support the interaction with and control of wirelessly interconnected devices, escaping the boundaries of current limited human computer interaction and enabling natural ambient interaction in smart environments. This opportunity on the other hand also creates new challenges for the user, who is faced with potential overload due to the need of starting, configuring, interconnecting, and using device ensembles in an appropriate and convenient way. In this paper, we contribute a novel approach for the automatic formation of appropriate ensembles of input and output devices to be used for novel ambient applications on- the-fly. We present a Fusion Engine as an extension of our Ambient Reflection Framework, which supports the dynamic and constrained-based setup of device ensembles by orchestrating a set of self-describing smart objects. The fusion takes into account the semantic information of applications, the services offered by smart objects and the physical constraints, users might face. We also discuss, how this process can be used to enable device ensembles to dynamically explain their functionality and interaction capabilities themselves to the user.

Human-Computer Interaction in Smart Environments

Here, we provide an overview of the content of the Special Issue on “Human-computer interaction in smart environments”. The aim of this Special Issue is to highlight technologies and solutions encompassing the use of mass-market sensors in current and emerging applications for interacting with Smart Environments. Selected papers address this topic by analyzing different interaction modalities, including hand/body gestures, face recognition, gaze/eye tracking, biosignal analysis, speech and activity recognition, and related issues.

Minotaurus: A System for Affective Human–Robot Interaction in Smart Environments

A service robot should be able to detect and identify the user in order to personalize its services and guarantee security, it should recognize the user’s emotions to allow affective interaction, and it should be able to communicate easily with the user and understand given commands by recognizing speech and gestures. Our research is motivated, for example, by the emerging needs of elderly care, health care, safety, and logistics. We have developed a distributed system for affective human–robot interaction which combines all the basic sensory elements and can collaborate with a smart environment and obtain further knowledge from the Internet. The realized HRI robot, Minotaurus, runs in real time and is capable of interacting with human beings. Minotaurus forms a rather generic platform for experimenting with human–robot collaboration in different applications and environments.

Affective Interaction in Smart Environments

We present a concept where the smart environments of the future will be able to provide ubiquitous affective communication. All the surfaces will become interactive and the furniture will display emotions. In particular, we present a first prototype that allows people to share their emotional states in a natural way. The input will be given through facial expressions and the output will be displayed in a context-aware multimodal way. Two novel output modalities are presented: a robotic painting that applies the concept of affective communication to the informative art and an RGB lamp that represents the emotions remaining in the user's peripheral attention. An observation study has been conducted during an interactive event and we report our preliminary findings in this paper.

Adaptive and context-aware privacy preservation exploiting user interactions in smart environments

In a pervasive system, users have very dynamic and rich interactions with the environment and its elements, including other users. To efficiently support users in such environments, a high-level representation of the system, called the context, is usually exploited. However, since pervasive environments are inherently people-centric, context might consist of sensitive information. As a consequence, privacy concerns arise, especially in terms of how to control information disclosure to other users and third parties. In this article, we propose context-aware approaches to privacy preservation in wireless and mobile pervasive environments. Specifically, we design two schemes: (i) to reduce the number of interactions between the user and the system; and (ii) to exploit the interactions between different users. Both solutions are adaptive and, thus, suitable for dynamic scenarios. In addition, our schemes require limited computational and storage resources. As a consequence, they can be easily implemented on resource-constrained personal communication and sensing devices. We apply our solutions to a smart workplace scenario and show that our schemes protect user privacy while significantly reducing the interactions with the system, thus improving the user experience.

Prediction of interface preferences with a classifier selection approach

Interaction in smart environments should be adapted to the users’ preferences, e.g., utilising modalities appropriate for the situation. While manual customisation of a single application could be feasible, this approach would require too much user effort in the future, when a user interacts with numerous applications with different interfaces, such as e.g. a smart car, a smart fridge, a smart shopping assistant etc. Supporting user groups, jointly interacting with the same application, poses additional challenges: humans tend to respect the preferences of their friends and family members, and thus the preferred interface settings may depend on all group members. This work proposes to decrease the manual customisation effort by addressing the cold-start adaptation problem, i.e., predicting interface preferences of individuals and groups for new (unseen) combinations of applications, tasks and devices, based on knowledge regarding preferences of other users. For predictions we suggest several reasoning strategies and employ a classifier selection approach for automatically choosing the most appropriate strategy for each interface feature in each new situation. The proposed approach is suitable for cases where long interaction histories are not yet available, and it is not restricted to similar interfaces and application domains, as we demonstrate by experiments on predicting preferences of individuals and groups for three different application prototypes: recipe recommender, cooking assistant and car servicing assistant. The results show that the proposed method handles the cold-start problem in various types of unseen situations fairly well: it achieved an average prediction accuracy of $$72 \pm 1\,\%$$ . Further studies on user acceptance of predictions with two different user communities have shown that this is a desirable feature for applications in smart environments, even when predictions are not so accurate and when users do not perceive manual customisation as very time-consuming.

Reconfigurable natural interaction in smart environments: approach and prototype implementation

The vision of sensor-driven applications that adapt to the environment hold great promise, but it is difficult to turn these applications into reality because device and space heterogeneity is an obstacle to interoperability and mutual understanding of the smart devices and spaces involved. Smart Spaces provide shared knowledge about physical domains and they inherently enable cooperative and adaptable applications by keeping track of the semantic relations between objects in the environment. In this paper, the interplay between sensor-driven objects and Smart Spaces is investigated and a device with a tangible interface demonstrates the potential of the $${\sl smart{\text -}space{\text -}based}$$ and $${\sl sensor{\text -}driven}$$ computing paradigm. The proposed device is named REGALS (Reconfigurable Gesture based Actuator and Low Range Smartifier). We show how, starting from an interaction model proposed by Niezen, REGALS can reconfigure itself to support different functions like Smart Space creation (also called $${\sl environment\,smartification}$$ ), interaction with heterogeneous devices and handling of semantic connections between gestures, actions, devices, and objects. This reconfiguration ability is based on the context received from the Smart Space. The paper also shows how tagged objects and natural gestures are recognized to improve the user experience reporting a use case and the performance evaluation of REGALS' gesture classifier.

Middleware mechanisms for supporting multimodal interactions in smart environments

Smart and intelligent environments can proficiently benefit a new generation of multimodal dialog systems, which enable users to interact in natural ways like voice, gesture, facial motions, and so on. However, in order to make this possible, it is necessary to integrate such systems and services, possibly in a transparent way. In the present paper, we propose an architectural model for multimodal smart environments; then, an agent-based middleware infrastructure that provides basic components for the integration of multimodal interaction systems; and, a set of ontologies and rules for semantically handling interacting properties of devices and resources available in smart environments.

Data Mining for Hierarchical Model Creation

In this paper, we examine the problem of learning inhabitant behavioral models in intelligent environments. We maintain that inhabitant interactions in smart environments can be automated using a data-driven approach to generate hierarchical inhabitant models and learn decision policies. To validate this hypothesis, we have designed the ProPHeT decision-learning algorithm that learns a strategy for controlling a smart environment based on sensor observation, power line control, and the generated hierarchical model. The performance of the algorithm is evaluated using real data collected from our MavHome smart home and smart office environments.