Chapter 6 - Regulating safety-critical autonomous systems: past, present, and future perspectives

Abstract

From unmanned aircraft to self-driving cars to closed-loop medical devices, autonomous systems offer great benefits but also pose new risks. Regulators must grapple with how to manage these risks, all while attempting to keep pace with technological developments and exhibit appropriate precaution without stifling innovation. Examination of how three agencies, the FAA, FDA, and NHTSA, regulate safety-critical systems, with a specific emphasis on their regulation of existing advanced automated systems, suggests important lessons learned that should be applied to the regulation of autonomous systems. These lessons include (1) regulation that begins early in a systems development life cycle may appear to be costly and time consuming, but for nascent technologies, systems engineering–based practices may provide critical checks and balances and ultimately such an approach may be the least expensive, (2) current regulation of automated safety-critical systems has not adequately addressed automation surprises in the form of mode confusion and software certification, which are likely to become more problematic with autonomous systems, and (3) expediting regulatory approvals for new technology with embedded software based on weak claims of existing equivalent technology likely significantly underestimates the risk that such technologies pose to the public. Going forward, not only do the federal agencies need to work together to better understand how regulation needs to fundamentally shift for autonomous systems but also need to consider companies and independent evaluators as critical stakeholders in defining future regulation.