Revisiting the issue of the credibility of simulation studies in telecommunication networks: highlighting the results of a comprehensive survey of IEEE publications

Abstract

In a series of papers by Pawlikowski and his team, they warned about a deep crisis of credibility in simulation studies for the performance evaluation of telecommunication networks. The goal of this article is to revisit the investigation on simulation credibility and to extend the previous research in this area by presenting the results of a comprehensive survey of over 8370 IEEE publications (2007?2009) on telecommunication networks which appeared in the proceedings of IEEE Transactions on Communications (TC), IEEE/ACM Transactions on Networking (TN), IEEE Infocom, IEEE ICC, and IEEE Globecom. These proceedings were selected based on their popularity and relevance to our study. Besides simulation credibility, we also report on the commonly used simulation tools and performance analysis methodology adopted by the authors of the surveyed papers. Our findings show that a significant number of authors did not provide information of the simulation tools used or, even when they did, there was not enough information about the CI or SE of simulation results presented in the surveyed papers. Thus, there is no significant change with respect to quality and credibility of the simulation studies revised and the deep crisis of credibility still remains. The research design most authors prefer is a hybrid one where both simulation and analytic methods are being used for network performance analysis. Finally, we provide guidelines for the best practice of using simulation tools for the performance evaluation of telecommunication networks.