A prediction interval-based approach to determine optimal structures of neural network metamodels

Abstract

Neural networks have been widely used in literature for metamodeling of complex systems and often outperform their traditional counterparts such as regression-based techniques. Despite proliferation of their applications, determination of their optimal structure is still a challenge, especially if they are developed for prediction and forecasting purposes. Researchers often seek a tradeoff between estimation accuracy and structure complexity of neural networks in a trial and error basis. On the other hand, the neural network point prediction performance considerably drops as the level of complexity and amount of uncertainty increase in systems that data originates from. Motivated by these trends and drawbacks, this research aims at adopting a technique for constructing prediction intervals for point predictions of neural network metamodels. Space search for neural network structures will be defined and confined based on particular features of prediction intervals. Instead of using traditional selection criteria such as mean square error or mean absolute percentage error, prediction interval coverage probability and normalized mean prediction interval will be used for selecting the optimal network structure. The proposed method will be then applied for metamodeling of a highly detailed discrete event simulation model which is essentially a validated virtual representation of a large real world baggage handling system. Through a more than 77% reduction in number of potential candidates, optimal structure for neural networks is found in a manageable time. According to the demonstrated results, constructed prediction intervals using optimal neural network metamodel have a satisfactory coverage probability of targets with a low mean of length.