Evaluating NASA Scientists’ Perspectives of the NASA Task Load Index

Abstract

Objective: The purpose of this research study was to evaluate NASA scientists' current perspectives of the NASA Task Load Index (NASA-TLX) and devise possible solutions for any noted limitations collaboratively. Background: The most commonly used self-report assessment tool for measuring workload is the NASA-TLX (Hart, 2006; Hart & Staveland, 1988). There have been many different variations of the NASA-TLX that are specifically tailored to the domain in which they are used (Hart, 2006; Wilson et al., 2011). These variations have been effective in identifying sources of workload in their environments (Wilson et al., 2013; Harris, Wilson, & Vine, 2020). The way the NASA-TLX is used in most research articles is as a dependent variable or comparison measure. As a dependent measure, it sufficiently identifies what independent variable of interest produced the most workload (Hart, 2006). Although the NASA-TLX has had widespread success, it does have limitations. In both research and applied settings, the current NASA-TLX is limited in its ability to identify environmental variables that contribute to workload. Method: We identified 25 scientists at NASA who were conducting work in human-robot operations who were familiar with the NASA-TLX and sent IRB-approved invitation letters to participate in a 14-item online survey. We received ten responses to the online survey. After the online survey, we conducted semi-structured phone interviews with five of the scientists. These scientists were selected to participate in phone interviews because they agreed to participate and indicated the NASA-TLX had practical limitations. They were also selected to participate in the phone interview if they reported having additional information they wanted to share or reported having attempted to modify the NASA-TLX in the past. Results: From the online surveys, we found that 80 percent of respondents reported that there were practical limitations with the NASA-TLX whereas 20 percent suggested there were not. Across the five individuals that we interviewed, we found similar responses. The scientists made statements such as: “We don’t necessarily get the time nor the funds to carry out studies that will require me to use a measure like the TLX”, “I would use it if it was an agile setting, it’s not necessarily the easiest to use. Now if there were an agile version of the TLX, maybe then” and “Beyond certain groups, people do not see quick value in using the NASA-TLX.” Conclusion: Based on the results of the survey and follow-up interviews, NASA scientists need a workload tool that is brief and has immediate practical applications. We have proposed an applied version of the NASA-TLX as a possible solution derived from this feedback. Application: This work has resulted in the development of an applied version of the NASA-TLX that may be able to identify sources of workload in various tasks in a shorter amount of time.