Sentient Computing Research Articles

A multimodal conversational coach for active ageing based on sentient computing and m‐health

AbstractAs life expectancy increases, it has become more necessary to find ways to support healthy ageing. A number of active ageing initiatives are being developed nowadays to foster healthy habits in the population. This paper presents our contribution to these initiatives in the form of a multimodal conversational coach that acts as a coach for physical activities. The agent can be developed as an Android app running on smartphones and coupled with cheap widely available sport sensors in order to provide meaningful coaching. It can be employed to prepare exercise sessions, provide feedback during the sessions, and discuss the results after the exercise. It incorporates an affective component that informs dynamic user models to produce adaptive interaction strategies.

Multi-sensory and Multi-modal Fusion for Sentient Computing

This paper presents an approach to multi-sensory and multi-modal fusion in which computer vision information obtained from calibrated cameras is integrated with a large-scale sentient computing system known as "SPIRIT". The SPIRIT system employs an ultrasonic location infrastructure to track people and devices in an office building and model their state. Vision techniques include background and object appearance modelling, face detection, segmentation, and tracking modules. Integration is achieved at the system level through the metaphor of shared perceptions, in the sense that the different modalities are guided by and provide updates to a shared world model. This model incorporates aspects of both the static (e.g. positions of office walls and doors) and the dynamic (e.g. location and appearance of devices and people) environment. Fusion and inference are performed by Bayesian networks that model the probabilistic dependencies and reliabilities of different sources of information over time. It is shown that the fusion process significantly enhances the capabilities and robustness of both sensory modalities, thus enabling the system to maintain a richer and more accurate world model.

Movement Awareness for Ubiquitous Game Control

This paper describes a sensing system for recognising and characterising human movements and its application to ubiquitous gaming. In particular this paper considers the control of computer games through players interacting with the physical environment around them in a natural and appropriate manner. This pushes the interface into the environment, and pulls ubiquitous computing into the game. This is achieved using a sentient computing system. Such a system senses the location, motions, actions and even physiological responses of users. This sensory data can be used to interface and control a game. A good example is 3D first-person games and we demonstrate a system in which actions in the game are mapped to similar actions in the real world.

TRIP: A Low-Cost Vision-Based Location System for Ubiquitous Computing

Sentient Computing provides computers with perception so that they can react and provide assistance to user activities. Physical spaces are made sentient when they are wired with networks of sensors capturing context data, which is communicated to computing devices spread through the environment. These devices interpret the information provided and react by performing the actions expected by the user. Among the types of context information provided by sensors, location has proven to be especially useful. Since location is an important context that changes whenever the user moves, a reliable location-tracking system is critical to many sentient applications. However, the sensor technologies used in indoor location tracking are expensive and complex to deploy, configure and maintain. These factors have prevented a wider adoption of Sentient Computing in our living and working spaces. This paper presents TRIP, a low-cost and easily deployable vision-based sensor technology addressing these issues. TRIP employs off-the-shelf hardware (low-cost CCD cameras and PCs) and printable 2-D circular markers for entity identification and location. The usability of TRIP is illustrated through the implementation of several sentient applications.

Implementing a sentient computing system

Sentient computing systems, which can change their behaviour based on a model of the environment they construct using sensor data, may hold the key to managing tomorrow's device-rich mobile networks. At AT&T Laboratories Cambridge, we have built a system that uses sensors to update a model of the real world. We designed the model's terms (object positions, descriptions and state, and so forth) to be immediately familiar to users. Thus, the model describes the world much as users themselves would. We can use this model to write programs that react to changes in the environment according to the user's preferences. We call this sentient computing because the applications appear to share the user's perception of the environment. Treating the current state of the environment as common ground between computers and users provides new ways of interacting with information systems. A sentient computing system doesn't need to be intelligent or capable of forming new concepts about the world, it only needs to act as though its perceptions duplicate the user's. In earlier work, we described a prototype of this system and stated our intention to deploy it on a large scale. We have now installed an enhanced version throughout an office building. Over the past year, approximately 50 staff members have used the system daily with a set of trial applications.