Measures of Real Time Assessment to use in Adaptive Augmentation

Abstract

Measures of Real Time Assessment to use in Adaptive Augmentation Martha E. Crosby (crosby@hawaii.edu) Curtis Ikehara (cikehara@hawaii.edu) David N. Chin (chin@ hawaii.edu) Department of Information and Computer Sciences, University of Hawaii 1680 East West Rd. POST 317, Honolulu HI 96822 Cognitive capabilities vary considerably due to people's different levels of expertise and aptitude. Also the cognitive capabilities of any one person will vary greatly over time because of stress, fatigue, injuries, attention lapses, and distractions. In order to augment the cognition of any individual, one strategy is to first assess the real-time cognitive capabilities of that individual and then tailor any augmentation to the current cognitive capabilities of the individual. The first major problem in real-time adaptive augmented cognition is assessing cognitive capabilities of the person instantaneously. A number of sensors have detected a student’s physiological states and actions. Riseberg et al. (1998) frustrated highly motivated users of a game interface by degrading mouse performance at irregular intervals. Galvanic skin conductivity, blood volume pressure, and muscle tension were recorded and correlated to frustration and non-frustration states. Ark, Dryer, and Lu (1999) established the feasibility of correlating physiological measurements to emotions. Crosby, Chin and Iding (2001) demonstrated how eye fixations can be used to analyze users’ strategies. Other research projects by Crosby, et al. (2001) suggest that a person's pressure on a mouse is also a good candidate for real- time biometrics to predict cognitive load. Identifying emotions is a complex process and it may not always be necessary to know the precise emotion. If other information such as performance data and focus of attention are available, it may be sufficient to only sense changes in the physiological data. The data collected by our projects include temperature, heart waveform, galvanic skin response and physical pressures applied to a mouse. Sensors are connected to an electrically isolated micro-controller, which converts the analog sensor signals to digital data for processing, by a master computer. The first sensor is on the finger tip to monitor peripheral temperature, the second sensor is attached to the wrist as a reference to body temperature and the third sensor monitors the ambient temperature at a distance of about 5 cm away from the wrist. The sensors are not attached to a substantial thermal mass that would reduce the response rate, and the minimum detectable change of temperature is calculated to be 0.005 C. The heart waveform is monitored using an infrared reflective light sensor that is mounted on the same elastic band as the fingertip temperature sensor. The change in blood flow is measured by changes in reflected light at the skin surface where blood flow is apparent. The heart beat rate is extracted by measuring the time between peaks and the relative blood volume change can be determined by integrating the heart waveform over the desired time. When the sensor is pressed against the fingertip, too much force will obstruct blood flow and too little force will produce unreliable data. The sensor also detects the mean reflected light, which is used to determine the optimum amount of pressure to be applied to the sensor. The experiments performed in our projects show that simple physiological measurements of GSR, heart rate, temperature, and pupil diameter provide information on changes in user’s emotional and subjective states while engaged in cognitive tasks. A subject’s physiological state and actions can be indicative of their cognitive performance. Physiological data, collected from people as they use computers to form their tasks, might not consistently predict their emotional state. However, results from our experiments suggest that it is possible to provide the computer with information about the users' cognitive state in real time. Acknowledgments This research was supported in part by Office of Naval Research grant no. N00014970578 and DARPA grant NBCH1020004. The authors thank to IBM’s project Blue Eyes for supplying two prototype emotion mice. References Ark, W., Dryer, D. and Lu, D. (1999). The Emotion Mouse. In Human-Computer Interaction: Ergonomics and User Interfaces, Bullinger. H. J. and J. Ziegler (Eds.), Lawrence Erlbaum Assoc., 818-823. Crosby, M., Auernheimer, B., Aschwanden, C., and Ikehara, C. (2001). Physiological Data Feedback for Application in Distance Education. Workshop on Perceptive User Interfaces, Florida. Crosby, M., and Chin, D. and Iding, M. (2001) Using Search Behavior in Complex Interfaces for the Design of Adaptive Systems Proceedings of 2001 User Modeling Conference, Germany. Riseberg, J., Klein, J., Fernandez, R., and Picard, R.W. (1998) Frustrating the user on purpose: Using biosignals in a pilot study to detect the user’s emotional state. CHI’98 Late- Breaking Results, 227-228 .