Algorithmengestützte Verwaltungsverfahren – Einsatzfelder, Risiken und Notwendigkeit ergänzender Kontrollen

Abstract

The digitization of the German public sector is gaining momentum and it now also extends to the use of algorithms and artificial intelligence in administrative procedures. On the basis of recently published official data, a broad spectrum of fields and variants of application can be identified. However, this increasing reliance on algorithms also creates particular challenges to the rule of law. In particular, the reliance on algorithms can increase the risk of systematic unlawfulness and bias in administrative decision-making, whether fully automated or still accounted for by a human official. Recent studies suggest that also in the latter case, those risks are often not properly managed – and are sometimes even exacerbated – by the humans in the loop. The traditional instruments of judicial protection and internal administrative supervision are not well suited to address the aforementioned inherent and operational risks either; moreover, they offer only ex post remedies and have no preventive effect. Therefore, additional technical precautions and institutional arrangements are required to uphold the rule of law. They should include investments in explainable artificial intelligence (“XAI”) and adequate staff training programs, as well as the creation of public registers and independent standard-setting and auditing bodies for public sector algorithms. In particularly sensitive areas of administration, the use of technical or proprietary black box algorithms should be entirely banned until further technological progress on transparency and risk control has been made.