Fault tolerance is a key requirement for En Route Automation Modernization (ERAM), the FAA's system that manages En Route air traffic over the USA. A system failure could lead to hundreds of flights being delayed or cancelled. Using experience from earlier systems a set of techniques were built into ERAM at inception, including a hot standby copy of each executable and the latest state checkpointed in disk files. As the system matured through formal testing and operational experience at the first sites (2010 - 2015), the goal of 100% availability was not achieved so additional techniques were added. These included exception safety, runaway process protection, and proactive monitoring of the system to detect defects and often resolve them without the air traffic controllers being aware. With the implementation of these additional techniques the FAA has measured ERAM as 100% available from October 2016 at all 20 operational sites. Software fault tolerance techniques have been well documented [2]; this extended abstract describes the specific techniques that have led to ERAM achieving continuous 24x7 availability for 6 years.
Read full abstract