Control-by-design? Autonomous weapons systems as technopolitical projects

In an influential paper Sparrow argues that it is immoral to deploy autonomous weapon systems (AWS) in combat. The general idea is that nobody can be held responsible for wrongful actions committed by an AWS because nobody can predict or control the AWS. I argue that this view is incorrect. The programmer remains in control when and how an AWS learns from experience. Furthermore, the programmer can predict the non-local behaviour of the AWS. This is sufficient to ensure that the programmer can be held responsible. I present a consequentialist argument arguing in favour of using AWS. That is, when an AWS classifies non-legitimate targets less often as legitimate targets, compared to human soldiers, then it is to be expected that using the AWS saves lives. However, there are also a number of reasons, e.g. risk of hacking, why we should still be cautious about the idea of introducing AWS to modern warfare.

The article explores autonomous weapon systems (AWS) operating with artificial intelligence as a complex challenge to contemporary international humanitarian law (IHL) and the international human rights framework. It analyses the technological capabilities and levels of autonomy of combat systems – including land, aerial, and naval unmanned platforms – that are already being used in current armed conflicts. Particular attention is given to the compliance of AWS with the core principles of IHL: distinction, proportionality, humanity, and the prohibition of indiscriminate attacks. The study substantiates the problem of «blurred» responsibility, particularly the difficulty of attributing violations committed by autonomous or semi-autonomous weapon systems to a specific accountable subject. It examines the risks posed by AWS to the observance of Articles 2, 3, 8, and 13 of the European Convention on Human Rights. The potential of the European Court of Human Rights’ case law to adapt to emerging technological realities through structured interpretation and the expansion of precedent is analysed. Special attention is given to international dialogue under the auspices of the United Nations – notably within the framework of the Convention on Certain Conventional Weapons (CCW), which addresses weapons deemed to cause excessive injury or have indiscriminate effects – and to the work of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons System, as well as the role of soft law, state positions, and international organisations. The article concludes with recommendations regarding the need to preserve meaningful human control, update legal mechanisms of responsibility, and develop a universal regulatory framework for AWS. It also highlights the importance of an interdisciplinary approach, particularly the integration of ethical, technical, and security considerations in shaping the legal regime governing the use of artificial intelligence systems and tools in military contexts.

Introduction This chapter examines the ethical reasons supporting a moratorium and, more stringently, a pre-emptive ban on autonomous weapons systems (AWS). Discussions of AWS presuppose a relatively clear idea of what it is that makes those systems autonomous. In this technological context, the relevant type of autonomy is task autonomy, as opposed to personal autonomy, which usually pervades ethical discourse. Accordingly, a weapons system is regarded here as autonomous if it is capable of carrying out the task of selecting and engaging military targets without any human intervention. Since robotic and artificial intelligence technologies are crucially needed to achieve the required task autonomy in most battlefield scenarios, AWS are identified here with some sort of robotic systems. Thus, ethical issues about AWS are strictly related to technical and epistemological assessments of robotic technologies and systems, at least insofar as the operation of AWS must comply with discrimination and proportionality requirements of international humanitarian law (IHL). A variety of environmental and internal control factors are advanced here as major impediments that prevent both present and foreseeable robotic technologies from meeting IHL discrimination and proportionality demands. These impediments provide overwhelming support for an AWS moratorium – that is, for a suspension of AWS development, production and deployment at least until the technology becomes sufficiently mature with respect to IHL. Discrimination and proportionality requirements, which are usually motivated on deontological grounds by appealing to the fundamental rights of the potential victims, also entail certain moral duties on the part of the battlefield actors. Hence, a moratorium on AWS is additionally supported by a reflection on the proper exercise of these duties – military commanders ought to refuse AWS deployment until the risk of violating IHL is sufficiently low. Public statements about AWS have often failed to take into account the technical and epistemological assessments of state-of-the-art robotics, which provide support for an AWS moratorium. Notably, some experts of military affairs have failed to convey in their public statements the crucial distinction between the expected short-term outcomes of research programmes on AWS and their more ambitious and distant goals. Ordinary citizens, therefore, are likely to misidentify these public statements as well-founded expert opinions and to develop, as a result, unwarranted beliefs about the technological advancements and unrealistic expectations about IHL-compliant AWS.

One of the several reasons given in calls for the prohibition of autonomous weapons systems (AWS) is that they are against human dignity (Asaro in Int Rev Red Cross 94(886):687–709, 2012; Docherty in Shaking the foundations: the human rights implications of killer robots, Human Rights Watch, New York, 2014; Heyns in S Afr J Hum Rights 33(1):46–71, 2017; Ulgen in Human dignity in an age of autonomous weapons: are we in danger of losing an ‘elementary consideration of humanity’? 2016). However there have been criticisms of the reliance on human dignity in arguments against AWS (Birnbacher in Autonomous weapons systems: law, ethics, policy, Cambridge University Press, Cambridge, 2016; Pop in Autonomous weapons systems: a threat to human dignity? 2018; Saxton in (Un)dignified killer robots? The problem with the human dignity argument, 2016). This paper critically examines the relationship between human dignity and AWS. Three main types of objection to AWS are identified; (i) arguments based on technology and the ability of AWS to conform to international humanitarian law; (ii) deontological arguments based on the need for human judgement and meaningful human control, including arguments based on human dignity; (iii) consequentialist reasons about their effects on global stability and the likelihood of going to war. An account is provided of the claims made about human dignity and AWS, of the criticisms of these claims, and of the several meanings of ‘dignity’. It is concluded that although there are several ways in which AWS can be said to be against human dignity, they are not unique in this respect. There are other weapons, and other technologies, that also compromise human dignity. Given this, and the ambiguities inherent in the concept, it is wiser to draw on several types of objections in arguments against AWS, and not to rely exclusively on human dignity.

A cybersystemic view of autonomous weapon systems (AWS)

Autonomous Weapon Systems and the Limits of Analogy

Critics of autonomous weapons systems (AWS) claim that they are both inherently unethical and unlawful under current international humanitarian law (IHL). They are unethical, it is said, because they necessarily preclude making any agent fairly accountable for the wrongful effects of AWS, and because allowing machines to make life or death decisions seriously undermines human dignity: only moral beings should make such decisions and only after careful moral deliberation, for which they could be held accountable. AWS are inherently unlawful, critics say, because they cannot possibly comply with the core IHL principles of discrimination and proportionality. Contrary to these critics, I argue in this chapter that AWS can conceivably be developed and deployed in ways that are compatible with IHL and do not preclude the fair attribution of responsibility, even criminal liability, in human agents. While IHL may significantly limit the ways in which AWS can be permissibly used, IHL is flexible and conventional enough to allow for the development and deployment of AWS in some suitably accountable form. Having indicated how AWS may be compatible with IHL and fair accountability, I turn to a serious worry that has been largely neglected in the normative literature on AWS. The development of AWS would deepen the already ongoing and very troubling dynamics of asymmetrical and so-called riskless warfare. While IHL-compatible AWS could be developed, in principle, and agents in charge of designing, testing and deploying AWS could be held accountable for wrongful harms, there are troublingly few incentives to duly control and minimize the risks to foreign civilians in the contexts of asymmetrical warfare. The most troubling aspects of AWS, I suggest, are not matters of deep ethical or legal principle but, rather, the lack of incentives for implementing effective regulations and accountability. The main goal of this chapter is to articulate this distinct worry and emphasize how serious it is. Once this is appreciated, it will be clear that more attention needs to be paid to determining what conditions would allow for the effective oversight of AWS development, testing and eventual use. Such oversight may be accomplished partly by defining liability criteria for agents working within the industrial and organizational complex behind AWS design, production and use. Ultimately, however, public scrutiny may be the only available effective push for IHL compliance and accountability.

Abstract: Critics of autonomous weapons systems (AWS) argue that these weapons cannot reliably distinguish between legitimate targets and those protected from attack. As a result, the use of AWS seems to violate the principle of distinction under international humanitarian law (IHL), which requires that combatants “not make civilians the object of attack” and not carry out attacks that are “indiscriminate in nature.” This criticism, however, misunderstands the principle of distinction and ignores important aspects of how AWS are being developed and deployed. Critics rely on an overly broad definition of AWS, and hold these systems to a standard that is inconsistent with the principle of distinction as it is actually formulated under IHL. Despite their very real limitations, the characteristics of modern AWS in fact highlight an impressive feat of technological design, and mark a further step on our long and fitful road to making warfare a less brutal and bloody enterprise.

This chapter identifies significant policy and military intersections between the evolving international cybersecurity and autonomous weapons systems (AWS) policy regimes that should receive deeper policy attention. So far, within policy discussions on lethal autonomous weapons systems (LAWS), there seems to have been less focus on related cyber implications compared with other policy questions. This is mirrored within the international cybersecurity policy community where AWS, maturing autonomous cyber technologies, and component technologies like artificial intelligence (AI) have not yet garnered extensive attention publicly. So far, most of the focus on AWS has centred on physical platforms for land, sea, air, space, and undersea, and not the cyber domain. Discussions surrounding AWS have generally been held under the rubric of the Convention on Conventional Weapons (CCW). Nevertheless, threat assessment reports and analysts are highlighting this subject more frequently. This chapter addresses these gaps by first unpacking the nature of so-called AWS and then highlighting a number of potential arms race considerations as well as consequences of the widespread adoption of autonomous technologies for warfare. It then proposes a framework to deal with the impact of autonomy on international security policies—namely strengthening technical safeguards and addressing the policy implications for international cyber stability. Lastly, the chapter argues for a need to ensure norm coherence and careful analysis of the implications arising from either banning or legitimizing maturing autonomous capabilities for international cybersecurity and AWS regimes.

There have been two recent strands of argument (one offered by Rob Sparrow and the other offered by Duncan Purves, Ryan Jenkins, and Bradley Strawser) arguing for the pro tanto impermissibility of fully autonomous weapon systems (AWS). On Sparrow's view, AWS are impermissible because they generate a morally problematic ‘responsibility gap’. According to Purves et al., AWS are impermissible because moral reasoning is not codifiable and because AWS are incapable of acting for the ‘right’ reasons. I contend that these arguments are flawed and that AWS are not morally problematic in principle. Specifically, I contend that these arguments presuppose an incoherent conception of an AWS as somehow making genuine decisions but not being morally responsible for those very same decisions. Rather than conceiving of AWS in this way, I argue that an AWS is either a socially‐constructed institution that has been physically instantiated or it is a genuine agent. If it is the former, then we should treat AWS as we do any other collective action problem. If it is the latter, then we should treat AWS as responsibility‐bearers, but also as bearers of rights and/or interests. To reject this disjunction is not only conceptually incoherent but also potentially morally dangerous.

As increasingly sophisticated weaponry reaches the field of battle, people are becoming increasingly isolated from the conflict. We already live in a world where a man sitting in a room can direct and carry out target-killing operations using robotic weapons on the opposite side of the globe. In this aspect, the advancement of weaponry technology has kept people off the battlefield, and the next step—artificial intelligence (AI) weapons—may do the same by removing people from decision-making. The use of AI technologies and techniques in warfare is growing quickly. This presented difficult difficulties to society, academics, lawmakers, military planners, and inventors. The development of AI weapons is already bolstering the armed markets; they are no longer the stuff of science fiction. Some nations have made significant progress in developing autonomous and machine learning systems from personnel systems, such as Israel's Iron Dome, which can stop approaching missiles autonomously and more quickly than a human could. President Putin stated to Russian students on September 8, 2017, that "artificial intelligence is the future, not only for Russia but for all of humankind.” Whoever assumes control of this arena will also assume control of the entire planet. An international discussion about whether and how such autonomous and machine learning weapons systems can conform with the standards of international humanitarian and customary law is being sparked by the development of AI weapons and technologies. The main issues in this paper are whether such autonomous weapons systems are effectively under human control, whether they can adhere to the fundamental principles of humanitarian law, such as distinction, proportionality, and the protection of civilians, what the nature of such armed conflict will be, and who will be held accountable for any mistakes. Finding the answers to those questions is the goal of this endeavour. The goal of this research is to briefly investigate the nature and character of warfare with AI weapons before outlining the significance and evolution of AI weapons. This study finishes by outlining the responsibilities under international humanitarian law that states may consider as part of their evaluations of weapons utilizing AI-related technologies.

Biases in artificial intelligence have been flagged in academic and policy literature for years. Autonomous weapons systems—defined as weapons that use sensors and algorithms to select, track, target, and engage targets without human intervention—have the potential to mirror systems of societal inequality which reproduce algorithmic bias. This article argues that the problem of engrained algorithmic bias poses a greater challenge to autonomous weapons systems developers than most other risks discussed in the Group of Governmental Experts on Lethal Autonomous Weapons Systems (GGE on LAWS), which should be reflected in the outcome documents of these discussions. This is mainly because it takes longer to rectify a discriminatory algorithm than it does to issue an apology for a mistake that occurs occasionally. Highly militarised states have controlled both the discussions and their outcomes, which have focused on issues that are pertinent to them while ignoring what is existential for the rest of the world. Various calls from civil society, researchers, and smaller states for a legally binding instrument to regulate the development and use of autonomous weapons systems have always included the call for recognising algorithmic bias in autonomous weapons, which has not been reflected in discussion outcomes. This paper argues that any ethical framework developed for the regulation of autonomous weapons systems should, in detail, ensure that the development and use of autonomous weapons systems do not prejudice against vulnerable sections of (global) society.

To many, the idea of autonomous weapons systems (AWS) killing human beings is grotesque. Yet critics have had difficulty explaining why it should make a significant moral difference if a human combatant is killed by an AWS as opposed to being killed by a human combatant. The purpose of this paper is to explore the roots of various deontological concerns with AWS and to consider whether these concerns are distinct from any concerns that also apply to long-distance, human-guided weaponry. We suggest that at least one major driver of the intuitive moral aversion to lethal AWS is that their use disrespects their human targets by violating the martial contract between human combatants. On our understanding of this doctrine, service personnel cede a right not to be directly targeted with lethal violence to other human agents alone. Artificial agents, of which AWS are one example, cannot understand the value of human life. A human combatant cannot transfer his privileges of targeting enemy combatants to a robot. Therefore, the human duty-holder who deploys AWS breaches the martial contract between human combatants and disrespects the targeted combatants. We consider whether this novel deontological objection to AWS forms the foundation of several other popular yet imperfect deontological objections to AWS.

We normally think that the autonomous weapons systems(AWS) and lethal autonomous weapons systems(LAWS), are one and the same, but I think that there is difference. Those who are opposing the AWS like ICRC, SIPRI and some countries are all emphasising a kind of moratorium on AWS, but they doesn’t differentiate the LAWS from AWS. While analysing their opinions, they actually meant that there is need for ban of AWS which going to take decisions on killing of human lives. The purpose of IHL is concerned with jus in bello not with jus ad bellum, and to maintenance of principle of proportionality in warfare, so the complete ban of AWS may not possible within purview of IHL. Hence we need to differentiate LAWS from AWS, and also to segregate LAWS from AWS. This paper going to work for that through technical and legal aspects using qualitative method and secondary data. KEY WORDS - Lethal Autonomous Weapon System , Conventional weapons, Iron dome, Loitering munitions, Harpy drone, principle of proportionality.

Autonomous systems will fundamentally alter the way wars are waged. In particular, autonomous weapon systems, capable of selecting and engaging targets without direct human operator involvement, represent a significant shift of humans away from the battlefield. As these new means and methods of warfare are introduced, many important targeting decisions will likely need to be made earlier and further away from the front lines. Fearful of these changes and coupled with other legal and moral concerns, groups opposed to autonomous weapons have formed and begun campaigning for a pre-emptive ban on their development and use. Nations intending to use these emerging technologies must grapple with how best to adjust their targeting processes and procedures to accommodate greater autonomy in weapon systems. This chapter examines these cutting-edge and controversial weapons with a particular emphasis on the legal impact on targeting during international armed conflicts. Initially, this chapter will explore the promising technological advances and operational benefits which indicate these weapon systems may become a reality in the not-so-distant future. The focus will then turn to the unique challenges the systems present to the law of armed conflict under both weapons law and targeting law principles. Next, the examination will shift to two key aspects of targeting most affected by autonomous systems: targeting doubt and subjectivity in targeting. The author ultimately concludes that autonomous weapon systems are unlikely to be deemed unlawful per se and that, while these targeting issues raise legitimate concerns, the use of autonomous weapons under many circumstances will be lawful.