Subjective Workload Assessment Research Articles

Unidimensional versus Multidimensional Workload Scales and the Effect of Number of Rating Scale Categories

The present study investigated why the Subjective Workload Assessment Technique (SWAT) (a multidimensional scale) has been found to be less sensitive than Overall Workload Scale (OW) (aunidimensional scale) in previous research. It was hypothesized that previous results could be attributed to number of points on the rating scale. To that end, four rating scale groups were employed in which the dimensionality (unidimensional vs multidimensional) and number of rating scale points (7 vs. 19) on the overall workload scale were varied. Forty-eight subjects performed a continuous recognition task composed of 12 levels of difficulty and then rated workload under one of four rating scale conditions (OW-7, OW-19, MOD-SWAT-7, MOD-SWAT-19). Results indicated that: (1) the preponderance of evidence favored equal sensitivity; (2) previous differences between OW and SWAT could not be due to differences in the number of points on the rating scale; and (3) dimensionality affected overall workload ratings in subtle ways. However, the differences found between the unidimensional (OW) and multidimensional (MOD-SWAT) scales is probably attributed to the nature of the task itself where increases in the difficulty in one dimension were accompanied by decreases in another dimension.

Information complexity-mental workload and performance in combat aircraft

The purpose of the present study was to analyse the effects of information complexity on Pilot Mental WorkLoad (PMWL) and Pilot Performance (PP), and to analyse the structure of PMWL. Eighteen pilots performed 72 simulated low level-high speed emissions. The complexity of the Head Down Display (HDD) information varied as a function of the tactical situation. Flight data were recorded continuously. The pilots' eye movements were video taped and psychophysiological activation data, Heart Rate (HR), were obtained. The pilots rated PMWL according to the psychological content of three scales (Bedford Rating Scale, Subjective Workload Assessment Technique, NASA-Task Load indeX) and answered a questionnaire tapping aspects of performance, information load, motivation and mood. It was found that even a moderate complexity of information interfered with the flight task. Altitude and variation in altitude were increased and corrections of altitude errors were delayed, when complexity increased. Changes in information load reached its maximum influence on flight performance (r = 0.60, p < 0.001) after a time delay of 20 to 40 seconds. Performance of the flight task correlated positively (r = 0.59, p < 0.001) with the performance of the information handling task (Tactical Situation Awareness, TSA). Durations and frequencies of eye fixations Head Up (HU) versus Head Down (HD) changed as a function of information load. A structural equation model implied that PMWL was affected by mission complexity and that PMWL affected objective and subjective aspects of flight performance and information handling. Heart rate (sortie means) correlated positively with PMWL (r = 0.34, p< 0.05) and perceived complexity of mission (r = 0.37, p < 0.01). Heart rate (running means) covaried with variations in information complexity for those pilots who performed well. From spectral analyses of cardiac interval. times it was found that the amplitude of the 0.10 Hz component tended to decrease during high as compared to low levels of information load.

NEW METHODS FOR SUBJECTIVE MENTAL WORKLOAD ASSESSMENT AND FUZZY RISK ANALYSIS

This paper presents new methods for subjective mental workload assess ment and fuzzy risk analysis in which simplified fuzzy number arithmetic operations are used for subjective mental workload assessment and fuzzy risk analysis The proposed methods can provide useful ways to handle subjective mental workload assessment problems and fuzzy risk analysis problems

Effects of control order, augmented feedback, input device and practice on tracking performance and perceived workload

Virtual interfaces to advanced human-machine systems will present operators with a variety of perceptual-motor challenges. To inform the virtual interface design processes, the present experiments examined the effects of track order, level of knowledge of performance, type of control device, and the extent of practice on tracking performance and associated mental workload. Tracking was assessed by root mean square error. Subjective workload was measured using both the NASA Task Load Index (TLX) and the Subjective Workload Assessment Technique (SWAT). Results indicated non-linear effects, where tracking error and subjective workload both increased non-proportionally with track order. Trackball use resulted in more accurate performance and was judged to be of lower subjective workload than input using a mouse. Augmented knowledge of performance had little effect on either performance or workload. There was a number of interactions affecting performance that were replicated in perceived workload. Over acquisition trials, second-order tracking exhibited continuous improvement. This capability was retained even after a 30-day rest interval. Decrease a workload followed performance improvement in both initial acquisition and subsequent retention phases. The two subjective workload scales were essentially equivalent in terms of their sensitivity to task manipulations. These results support the direct association between workload and performance and confirms the use of workload in helping to evaluate the influence of diverse task-related demands. The implications for the design of virtual interfaces to real-world systems are examined in the light of these findings.

Diagnosticity and multidimensional subjective workload ratings.

A new multidimensional subjective workload assessment instrument -- Workload Profile -- was introduced and evaluated against two unidimensional instruments -- Bedford and Psychophysical scaling. Subjects performed two laboratory tasks separately (single task) and simultaneously (dual task). The multidimensional procedure compared well with the unidimensional procedures in terms of sensitivity to task demands, concurrent validity with performance, and test-retest reliability. The results suggested that the subjective workload profiles would only have limited predictive value on performance. However, results of the canonical analysis demonstrated that the multidimensional ratings provided diagnostic information on the nature of task demands. Further, the diagnostic information was consistent with the a priori task characterization. This strongly supports the notion that mental workload is multidimensional and that subjects are capable of reporting the demands on separate workload dimensions. Theoretical implications on mental workload models and practical implications on the assessment approaches are discussed.

Chapter 8 Cardiovascular and Respiratory Responses During a Complex Decision-Making Task Under Prolonged Isolation

Self-administered psychophysiological assessment during a mission is not only feasible, but yields reliable data that can be related to mission demands in a meaningful way. The unique methodology that has been employed during the EXEMSI campaign, may be of great promise for the development of techniques that provide daily monitoring of physiological cost of mission demands. This approach can thus be used for obtaining a better understanding of problems that can arise with regard to the functioning of individual crew members. The response profiles of the individual crew members showed remarkable quantitative and qualitative differences during isolation. Due to the limitations of this study, these differences could by no means all be explained in terms of mission demands, operator workload or other relevant parameters. However, the findings do seem to permit some tentative conclusions. In subjects B and D the cardiovascular response profiles suggested that cumulative stress effects emerged, in particular during the second part of the mission. It is important to note that both subjects carried specific responsibilities for the success of the mission, and that there was evidence for a conflict between them. Concerning the question which physiological measures are most appropriate for in-mission psychophysiological assessment, these results can appear to imply that cardiovascular measures yield more valid information about the effects of mission demands that respiratory measures. However, it should be noted that important aspects of respiratory activity, namely, respiratory volume measures and ventilation, could not be reliably determined in this study. Therefore, definitive conclusions concerning the choice of measures wait until the validity of the measures has been more extensively evaluated. Although the methodology that was developed for this study appears to be promising, it is obvious that its value can only be properly assessed when the physiological findings are correlated with behavioral, performance and subjective data. An extensive analysis of the covariation of the daily variations in physiological response, performance measures, and subjective assessments of workload and mood, is currently in progress in collaboration with Hockey and Sauer. Finally, if the usefulness and validity of this type of in-mission self-administered psychophysiological assessment are confirmed in ongoing and future studies (e.g., the 1994 HUBES mission), protocols will have to be worked out for practical implementation during actual spaceflights. In this regard, it has been advocated that psychophysiological monitoring techniques should include feedback and support policies, make crew members aware of potential risks for breakdown, and enable them to take appropriate measures when necessary. Initially, this can perhaps best be realized by procedures which include a data-link between the spacecraft and ground mission control. On the basis of ground-based analysis and evaluation of the data, psychologists at mission control may decide to provide information to the crew concerning potential individual or intragroup overload or stress problems, and to implement specific supportive measures. The effectiveness of the supportive activities should be closely monitored by mission control. Future work should also be aimed at developing expert-systems, which will enable the crew to maintain an appropriate degree of inflight biobehavioral self-regulation by providing a range of support techniques, like biofeedback, coping techniques against individual and group stressors, work/rest and sleep/wake schedules, and emergency procedures.

SWAT: Do We Need Conjoint Measurement?

The present study sought to determine the utility of the unweighted sum as an alternative to conjoint measurement in forming a workload composite measure with the Subjective Workload Assessment Technique (SWAT). The data from a simulator study designed to identify pilot workload associated with alternative cockpit configurations in a night air interdiction mission were reanalyzed using a percentage measure (SUMPCT) based on the unweighted sum of the three SWAT rating scales. Results indicated that the CONJOINT and SUMPCT workload composites were highly correlated and were equally sensitive to detecting differences in the independent variables. The reasons for the equal sensitivity of the two composite measures are discussed. This study adds to the growing body of evidence that one can form a composite measure of workload using the SWAT rating scales without the necessity of having to conduct the time-consuming card sort.

Some Measurement and Methodological Considerations in the Application of Subjective Workload Measurement Techniques

Previous research with the NASA Task Load Index (TLX) and the Subjective Workload Assessment Technique (SWAT) has demonstrated their sensitivity to variations in workload under flight and other complex work conditions but has not addressed some important measurement and methodological issues regarding their application. This article reviews data that address several such issues including the effects of delayed workload ratings, the effects of previous workload levels on ratings, and procedures used to combine subscale ratings into one overall estimate of subjective workload.

Mental workload and cognitive task automaticity: an evaluation of subjective and time estimation metrics

The evaluation of mental workload is becoming increasingly important in system design and analysis. The present study examined the structure and assessment of mental workload in performing decision and monitoring tasks by focusing on two mental workload measurements: subjective assessment and time estimation. The task required the assignment of a series of incoming customers to the shortest of three parallel service lines displayed on a computer monitor. The subject was either in charge of the customer assignment (manual mode) or was monitoring an automated system performing the same task (automatic mode). In both cases, the subjects were required to detect the non-optimal assignments that they or the computer had made. Time pressure was manipulated by the experimenter to create fast and slow conditions. The results revealed a multi-dimensional structure of mental workload and a multi-step process of subjective workload assessment. The results also indicated that subjective workload was more influenced by the subject's participatory mode than by the factor of task speed. The time estimation intervals produced while performing the decision and monitoring tasks had significantly greater length and larger variability than those produced while either performing no other tasks or performing a well practised customer assignment task. This result seemed to indicate that time estimation was sensitive to the presence of perceptual/cognitive demands, but not to response related activities to which behavioural automaticity has developed.

The Roles of Immediacy and Redundancy Relative Subjective Workload Assessment

Subjective workload assessment is one of the most frequently used tools for system evaluation. However, little is known about which factors contribute to the sensitivity, validity, and reliability of subjective workload ratings. One major classification of subjective workload assessment tools is based on relative judgments of the workload of different task conditions. The present research evaluated different approaches to relative subjective workload assessment in terms of their sensitivity to demand manipulations, concurrent validity with performance, and test-retest reliability. Results indicated that performing relative judgments retrospectively after having experienced all task conditions was superior to rating each task condition immediately after performing it. Further, redundant relative comparisons may produce more sensitive ratings than do relative comparisons to a single reference task. Overall, the results support the use of retrospective relative workload judgments as an evaluation tool.

Subjective assessment of mental workload — A fuzzy linguistic multi-criteria approach

In this paper, a fuzzy linguistic multi-criteria method for subjective assessment of mental workload is proposed. The criteria ratings and the corresponding importance weights are assessed in linguistic terms, which are described by fuzzy numbers with triangular and trapezoidal membership function. A fuzzy weighted average algorithm, based on the extension principle, is employed to aggregate these fuzzy numbers. The aggregated fuzzy number is then translated back to linguistics and the overall mental workload is expressed in linguistic terms.

An objective methodology for task analysis and workload assessment in anesthesia providers.

Administering anesthesia is a complex task in which either human or equipment failure can have disastrous consequences. An improved understanding of the nature of the anesthesiologist's job could provide a more rational basis for improvements in provider training as well as the design of anesthesia equipment. The objective of this study was to develop a set of techniques to evaluate anesthesiologist performance and to determine what information could be obtained from performing real-time task assessment and workload analysis tests in the operating room. The methodology used included time-motion analysis, secondary task probing, and subjective workload assessment. The time-motion data was subjected to subsequent analysis to generate quantitative measures such as task duration (time spent focused on an individual task) and task density (the number of tasks initiated per minute). The latency of response to a "vigilance light" was used as a secondary task probe. Finally, both the observer and the subjects themselves scored workload at 10-15-min intervals throughout the case. Two groups of anesthesia providers performing general endotracheal anesthesia for simple ambulatory surgical cases (1-4 h duration) were examined using this methodology. In the first group, 3rd-yr anesthesia residents and experienced certified registered nurse anesthetists (n = 11) performed cases under limited supervision by an attending anesthesiologist. In the second group, novice residents in their first 8 weeks of training (n = 11) performed similar cases under nearly constant attending supervision. The two groups seemed to manifest different patterns of task behavior, task density, subjective workload, and latency of response to the vigilance task. Response latency to the vigilance task increased at times of increased workload (e.g., during induction of anesthesia). The experienced (less supervised) providers spent significant amounts of time observing the monitors and the surgical field, whereas the novice subjects spent more time conversing with the supervising attending. Despite performing fewer tasks per minute (lower task densities), the novice subjects exhibited longer latencies of response to the vigilance light and increased subjective workload. Novice subjects also had longer task duration values. For example, postintubation, novices focused on their monitors for an average of twice as long as did experienced subjects (13 +/- 2 vs. 7 +/- 1 s) before moving on to another task. These techniques permitted an objective description of task characteristics, workload, and vigilance in anesthesia personnel under actual work conditions. This methodology could aid in understanding the factors that affect anesthesiologists' performance and may prove useful in assessing the progress of training.

Measuring Subjective Workload: When Is One Scale Better Than Many?

The idea that mental workload should be treated as a multidimensional problem is well recognized. Yet the NASA Task Load Index and Subjective Workload Assessment Technique scales incorporate procedures for combining multidimensional judgments into scalar estimates of overall workload. In view of comparable or superior performance claimed for univariate scales, it is reasonable to ask whether the scaled values are good estimates of overall processing demands and whether workload values derived from multidimensional scales provide better estimates than do judgments made on a univariate scale. This paper examines these issues using data from four independent studies. The results support two conclusions. First, if a measure of the overall demand on human information-processing is required, then a univariate rating is expected to provide a measure that is at least as sensitive to manipulations of task demand as a derived estimate from multivariate data. Second, if a univariate workload rating is not available, a simple unweighted additive model provides an adequate method for combining factor ratings into an estimate of overall workload.

Black clouds. Work load, sleep, and resident reputation.

Although it is assumed that residents in a specific training program will have comparable experiences, residents commonly perceive that some have consistently more difficult times on call. Such residents in our program are said to have "black clouds." We sought to determine if these perceptions were related to differences in real work load. We collected data about the on-call experiences of our first-year pediatric residents (PL-1s) for 358 days (1355 on-call experiences) during the 1984-1985 academic year. Every PL-1 (n = 19) reported the following data the morning after each night on call: hours of sleep, number of admissions, total number of patients, number of deaths, number of transfers to the pediatric intensive care unit, number of delivery room trips, and a subjective assessment of work load, using a three-point scale. The reputation of each house officer was determined by asking all residents in the program (PL-1s, PL-2s, and PL-3s) to rate each other three times during the year regarding how hard they worked on call. There were significant differences among PL-1s in how difficult they perceived their work load to be and in how much they slept (P < .001 using analysis of variance). However, actual work load (as measured by the number of either admissions or patients) did not vary significantly among the residents. There was a strong negative association between self-perception of work load and hours of sleep (r = -.75; 95% confidence interval, -0.73 to -0.76). Sleep was the major predictor of perceived work load (multiple R2 = .563 using multiple linear regression analysis). The absence of an association between perceived and actual work load is attributed to large differences in the residents' working styles. This is evidenced by a wide range of correlations among PL-1s between the number of admissions and hours of sleep (range of r values, -.66 to -.16). A reputation for difficult on-call experiences was strongly associated with few hours of sleep (r = -.77; 95% confidence interval, -0.49 to -0.91), but not with actual work load measured by the number of admissions, patients, deaths, or other variables. Sleep was the major predictor of reputation (multiple R2 = .567 using multiple linear regression analysis). Some residents did have a black cloud; they slept less, perceived that they worked harder than average, and had a reputation for having difficult on-call experiences. Residents with a black cloud function differently from their colleagues; for example, some may be inefficient, while others may create extra work for themselves. Residency program directors must recognize these functional differences to effectively evaluate and counsel house officers.

Establishing Workload Acceptability: An Evaluation of a Proposed Kc-135 Cockpit Redesign

Workload assessment has become a common part of system evaluation. Workload assessment is an important adjunct to performance measurement because the operator is sometimes flexible enough to disguise excessively demanding systems by expending additional effort to overcome optimal information processing limits. This is often referred to as the problem of determining a “workload redline.” The present paper recounts an evaluation of a proposed redesign of the KC-135 tanker aircraft cockpit. The current KC-135 cockpit has three crew positions: pilot, copilot, and navigator. As part of a proposed redesign, modern automation capabilities to replace the navigator were considered. Ten operational KC-135 crews and two KC-10 crews were studied while performing missions of differing levels of workload in a high-fidelity simulator. Three main classes of data relevant to the redline issue were collected: Performance data. Subjective Workload Assessment Technique (SWAT) ratings, and Subjective WORkload Dominance (SWORD) ratings. Evaluation of the performance results demonstrated that the redesigned cockpit could be flown in accordance to regulations. This was a necessary first step, but could not ensure that acceptable workload had been obtained. Taken together, the SWAT and SWORD results strongly suggested that acceptable performance can be achieved at acceptable level s of workload. In conclusion, the present study is a prototypical example of using available assessment tools to determine system acceptability. These tools should be useful for many other system evaluations.

Effects of Aging on Subjective Workload and Performance

Demographics indicate that the population in the United States and other industrialized nations is growing older, and that the number of older workers and systems users can be expected to increase substantially over the next several decades. In order to assess possible differences between age groups the mental workload experienced by older adults as compared to that experienced by younger adults was investigated. Two tasks were utilized to assess short term memory (continuous recognition) and psychomotor (first-order unstable tracking) performance. The workload of each task was assessed with the Subjective Workload Assessment Technique (SWAT). Memory task performance measures and subjective workload ratings indicated a decrement in performance and an increase in workload for the older group relative to the younger group. Psychomotor task performance measures and subjective workload ratings indicated no difference between the age groups. It is hypothesized that the memory task makes greater demands on central processing resources than the psychomotor task used in this study. In support of this hypothesis, an analysis of the changes in ratings on the individual SWAT dimensions of time, mental effort and psychological stress revealed that an increase occurred only on the mental effort dimension for the memory task. This study implies that designers should 1) reduce or provide design features that lessen memory laden task performance for older workers, and 2) give more weight to the reduction of central processing resource requirements in trade-off studies.

Application of the Subjective Workload Assessment Technique to Aviation Test and Evaluation

A study was performed to assess pilot workload associated with the employment of an air-to-air weapon system integrated onto an attack helicopter. Mental workload was assessed using the Subjective Workload Assessment Technique (SWAT). Pilots performed simulated engagements against an airborne target under varying conditions of engagement type, time of day, target background, and target range. The results indicated significant differences in SWAT ratings as a function of time of day and engagement type. To a lesser degree, SWAT ratings were also sensitive to changes in target background and range. These results are consistent with laboratory and simulation studies which have shown SWAT to be sensitive to changes in task demand and further demonstrate the utility of SWAT for assessing operator workload in the less structured test and evaluation environment.

Comparison of Four Subjective Workload Rating Scales

Four subjective workload scales were compared along four dimensions: sensitivity, operator acceptance, resource requirements, and special procedures. The scales were the Modified Cooper-Harper scale, the National Aeronautics and Space Administration Task Load Index (TLX), the Overall Workload (OW) scale, and the Subjective Workload Assessment Technique. Three U.S. Army systems were studied for potential workload concerns. Data from five different studies on the three systems were compared along the aforementioned four dimensions. Results indicate that all four scales are acceptable tools and are sensitive to different levels of workload. However, TLX and OW are consistently superior when considering sensitivity, as measured by factor validity, and operator acceptance. This research is an example of a systematic approach for examining human factors measurement tools.

In-Flight and Postflight Assessment of Pilot Workload in Commercial Transport Aircraft Using the Subjective Workload Assessment Technique

A question relevant for Federal Aviation Administration (FAA) certification of pilot workload in new commercial transport aircraft is whether subjective evaluation of workload should be conducted in-flight or postflight. This article examines the effect of probe timing (in-flight or postflight) for subjective assessment of flights containing both low and high levels of workload using the Subjective Workload Assessment Technique. The B-727, FAA Phase 2-certified, 6-dr, motion-base simulator, and air traffic control simulation at the Man-Vehicle System Research Facility at the NASA-Ames Research Center were used. Each pilot flew three different flights: (a) a nominal flight (30 rain); (b) a communication flight in which the pilot was instructed to tune in and talk on the command radio, in addition to normal duties (30 rain); and (c) a malfunction flight, including two diversions and an engine and "A" hydraulic system failures (1 hr 30 min). Sixteen B-727 qualified airline pilots flew the three flights on two separate occasions in a test-retest paradigm. No effect for probe timing (in-flight or postflight) was found for the nominal and communication flights. A significant (Probe Timing x Phase of Flight) interaction was found for the malfunction flight. It appears that postflight ratings of high-workload events receive higher workload ratings than in-flight ratings of the same event.

Testing a Subjective Metric of Situation Awareness

The present research tested a subjective situation awareness (SA) metric, called the SA-SWORD technique. The SA-SWORD technique was an SA adaptation of a subjective workload assessment tool (called the Subjective WORkload Dominance (SWORD) technique). The utility of the SA-SWORD technique was tested in a simulator evaluation of cockpit displays, using Air Force pilots as subjects. Two displays were compared: the Fire Control Radar (FCR) display, and the Horizontal Situation Format (HSF) display. The FCR display provided a functional simulation of current air-to-air radar displays. The HSF was a map-like display that combined data-linked information from outside sources (e.g., an AWACS) with the aircraft's own data to provide a bird's-eye, symbolic view of the entire area. Following collection of the performance data, SA-SWORD scores were collected. Pilots rated their SA higher when using the HSF display, than when using the FCR display. An implication of the current results is that subjective SA metrics can play a useful role in a system evaluation. More specifically, the current results suggest that the SA-SWORD technique can be a useful tool for assessing subjective SA.