Team performance in a series of regional and national US cybersecurity defense competitions: Generalizable effects of training and functional role specialization

Abstract

A critical component to any modern cybersecurity endeavor is effective use of its human resources to secure networks, maintain services and mitigate adversarial events. Despite the importance of the human cyber- analyst and operator to cybersecurity, there has not been a corresponding rise in data-driven analytical approaches for understanding, evaluating, and improving the effectiveness of cybersecurity teams as a whole. Fortunately, cyber defense competitions are well-established and provide a critical window into what makes a cybersecurity team more or less effective. We examined data collected at the national finals and four regional events of the Collegiate Cyber Defense Competition and posited that experience, access to simulation-based training, and functional role composition by the teams would predict team performance on four scoring dimensions relevant to the application of information assurance skills and defensive cyber operations: (a) maintaining services, (b) help-desk customer support, (c) handling scenario injects, and (d) mitigating red team attacks. Bayesian analysis highlighted that experience was a strong predictor of service availability, scenario injects, and red team defense. Simulation training was also associated with good performance along these scoring dimensions. High-performing and experienced teams clustered with one another based on the functional role composition of team skills. These results are discussed within the context of stages of team development, the efficacy of challenge-based learning events, and reinforce previous analytical results from cyber competitions.