Examining the Effects of Implementing Data-Driven Uncertainty in Cost Estimating Models

Abstract

When conducting probabilistic cost analysis, correlation assumptions are key assumptions and often a driver for the total output or point estimate of a cost model. Although the National Aeronautics and Space Administration (NASA) has an entire community dedicated to the development of statistical cost estimating tools and techniques to manage program and project performance, the application of accurate and data-driven correlation coefficients within these models is often overlooked. Due to the uncertain nature of correlation between random variables, NASA has had difficulty quantifying the relationships between spacecraft subsystems with specific, data-driven correlation matrices. Previously, the NASA cost analysis community has addressed this challenge by either selecting a blanket correlation value to address uncertainty within the model or opting out of using any correlation value altogether. One hypothesized method of improving NASA cost estimates involves deriving subsystem correlation coefficients from the residuals of the regression equations for the cost estimating relationships (CERs) of various spacecraft subsystems and support functions. This paper investigates the feasibility of this methodology using the CERs from NASA's Project Cost Estimating Capability (PCEC) model. The correlation coefficients for each subsystem of the NASA Work Breakdown Structure were determined by correlating the residuals of PCEC's subsystem CERs. These correlation coefficients were then compiled into a 20x20 correlation matrix and were implemented into PCEC as an uncertainty factor influencing the model's pre-existing cost distributions. Once this correlation matrix was implemented into the cost distributions of PCEC, the Latin Hypercube Sampling function of the Microsoft Excel add-in Argo was used to simulate PCEC results for 40 missions within the PCEC database. These steps were repeated three additional times using the following correlation matrices: (1) a correlation matrix assuming the correlation between each subsystem is zero, (2) a correlation matrix assuming the correlation between each subsystem is 1, and (3) a correlation matrix using a blanket value of 0.3. The results of these simulations showed that the correlation matrix derived from the residuals of the subsystem CERs significantly reduced bias and error within PCEC's estimating capability. The results also indicated that the probability density function and cumulative distribution function of each mission in the PCEC database were altered significantly by the correlation matrices that were implemented into the model. This research produced (1) a standard subsystem correlation matrix that has been proven to improve estimating accuracy within PCEC and (2) a replicable methodology for creating this correlation matrix that can be used in future cost estimating models. This information can help the NASA cost analysis community understand the effects of applying uncertainty within cost models and perform sensitivity analyses on project cost estimates. This is significant because NASA has been frequently critiqued for underestimating project costs and this methodology has shown promise in improving NASA's future cost estimates and painting a more realistic picture of the total possible range of spacecraft development costs.