Please notice that this is a DRAFT of a paper the final version of which has been published as Fossa, F. (2018). Legal Fictions and the Essence of Robots: Thoughts on Essentialism and Pragmatism in the Regulation of Robotics. In M. Coeckelbergh, J. Loh, M. Funk, J. Seibt, & M. Nørskov (Eds.), Envisioning Robots in Society – Power, Politics, and Public Space, Frontiers in Artificial Intelligence and Applications, Vol. 311, IOS Press, pp. 103-111 and is available for purchase at the following link: http://ebooks.iospress.nl/volumearticle/50788 Legal Fictions and the Essence of Robots: Thoughts on Essentialism and Pragmatism in the Regulation of Robotics Fabio FOSSAa,1 a Institute of Law, Politics, and Development, Sant'Anna School of Advanced Studies, Italy. Abstract. The purpose of this paper is to offer some critical remarks on the so-called pragmatist approach to the regulation of robotics. To this end, the article mainly reviews the work of Jack Balkin and Joanna Bryson, who have taken up such approach with interestingly similar outcomes. Moreover, special attention will be paid to the discussion concerning the legal fiction of 'electronic personality'. This will help shed light on the opposition between essentialist and pragmatist methodologies. After a brief introduction (1.), in 2. I introduce the main points of the methodological debate which opposes pragmatism and essentialism in the regulation of robotics and I examine how legal fictions are framed from a pragmatist, functional perspective. Since this approach entails a neat separation of ontological analysis and legal reasoning, in 3. I discuss whether considerations on robots' essence are actually put into brackets when the pragmatist approach is endorsed. Finally, in 4. I address the problem of the social valence of legal fictions in order to suggest a possible limit of the pragmatist approach. My conclusion (5.) is that in the specific case of regulating robotics it may be very difficult to separate ontological considerations from legal reasoning-and vice versa-both on an epistemological and social level. This calls for great caution in the recourse to anthropomorphic legal fictions. Keywords. Essentialism, pragmatism, legal fictions, electronic personality, anthropomorphism 1. Introduction The purpose of this paper is to offer some (admittedly partial and non-systematic) remarks on the so-called pragmatist approach to the regulation of robotics as opposed to the essentialist approach. In order to do that, I will mainly focus on the work of Jack Balkin and Joanna Bryson, who have taken up the pragmatist approach with interestingly similar outcomes. Special attention will also be paid to the question concerning the role that legal fictions such as the 'electronic personality' may play in this scenario. Since pragmatists frame this problem in functional and anti-essentialist terms, to see whether such terms can actually be met will hopefully put the study on the right track. After introducing the main arguments of the methodological debate that opposes pragmatism and essentialism in the regulation of robotics, I will discuss Balkin's and Bryson's suggestions on 'robotic' legal fictions and 'electronic personality' by asking three questions: 1 Fabio Fossa, via Luigi Borghi 11, 21013 Gallarate (VA) Italy. E-mail: fabio.fossa@santannapisa.it • From a pragmatist perspective, when is it appropriate to introduce legal fictions? (section 2.) • Can essentialist considerations be entirely put into brackets? (section 3.) • Is legal convenience the only criterion needed to decide whether to introduce the 'electronic personality' fiction? (section 4.) 2. Legal Fictions as Functional Regulatory Constructs 2.1. Robots, Legal Fictions, and Law's Ends Let me start by sketching the methodological background of the topic I wish to address. As Jack Balkin [1] explains, the many issues concerning the regulation of robotics are usually addressed in two opposite ways, which he labels "essentialism" and "pragmatism". In a nutshell, the advocators of essentialism argue that legal frameworks for robotics should be tailored on ontological analyses of robots' essential qualities or capabilities [24]. In other words, it is maintained that the first step towards the proper regulation of robotics is to determine what a robot is and is not-and, more importantly, what a robot can and cannot do. This not only requires a certain degree of scientific knowledge of the technology involved, but also relies on philosophical discussions to give meaning and context to it. In sum, according to this view the legal task is supposed to be based on an ontological study into the essence of robots. In The Path of Robotic Law, Balkin highlights two main limitations of essentialism [1, p. 45]. First, due to the inner vitality of robotic research and the ever-changing ways in which new technologies impact people's lives, it appears quite difficult, if not impossible, to define what a robot is in a way that can at the same time be accurate, informative, and of some use for regulatory purposes [5, pp. 217f.]. Secondly, and more interestingly, the connection between ontological analysis and legal reasoning has not been traditionally seen as binding. This is true, in particular, when proposals for the attribution of rights and obligations to non-human entities are advanced–as notably happens with robots (see infra). Grounding regulatory measures on ontological premises is risky, as it could potentially lead to a framework too rigid to properly address the ever-changing ways in which technologies modify our habits. In this situation, then, it may be better to adopt a more pragmatic attitude. Accordingly, Balkin suggests readdressing attention from robots' essence to how their uses transform specific aspects of social practices which are valuable for and can be regulated by law. "When we consider how a new technology affects law," Balkin writes, "our focus should not be on what is essential about the technology but on what features of social life the technology makes newly salient" in relation to law's ends [1, p. 46]. This claim, as the author points out [1, p. 46], is grounded on a functional theory of law, according to which legal systems are to be understood by the social function they carry out or the social ends they serve. Consequently, a good legal framework for robotics would be one which served law's ends properly. Once such ends are spelled out, then, what is left is an instrumental task: means are to be devised to achieve given ends. In carrying out this task, lawmakers should be left free to abstract from ontological considerations. For example, they should be left free to turn to legal fictions-artificial constructs which carry no ontological commitment but work as useful means to ends which are valuable in the eyes of the law [1, p. 59]. This, in fact, is merely a technical matter and bears no implication outside the legal domain. Ontology, on the contrary, would unnecessarily tie law's hands. Therefore, it should be kept out of the picture. 2.2. Legal Personality for Robots? A similar line of reasoning is applied in Of, For, and By the People by Joanna Bryson, Mihailis Diamantis, and Thomas Grant [6] in order to discuss the proposition of granting some form of legal personality to robots as a regulatory tool [7, 8]. This attribution would be fictional, meaning that it would neither be based on ontological analysis nor entail any ontological claims; it would rather be a purely legal act, entirely independent from ontological accounts of such entities as natural persons. The purpose of this act-i.e., to help allocate responsibilities, uphold rights, and determine obligations in concrete situations [9, pp. 18f.]-would also be of a purely legal nature. Hence, it would be an error to oppose the attribution of legal personality to robots on ontological grounds. The matter must be discussed exclusively in legal terms [5, pp. 242f]. In Bryson's and colleagues' view, legal systems are social products whose purpose is to promote the material interests of legal persons, to protect important moral rights and to enforce relevant moral obligations–especially and primarily those of human beings [6, pp. 282-283]. In this scenario, the debate on electronic personality becomes "ultimately a pragmatic question-Does endowing robots with this legal right or that legal obligation further the purposes of the legal system?" [6, p. 282].2 The authors' answer to such pragmatic question is that the costs of attributing legal personality to artificial agents would overcome the benefits. It is unclear, in fact, how robots endowed with legal personality would respond to accusations or would be properly punished for breaking the law or falling short of their obligations. All things considered, attributing legal personality to robots may turn out to facilitate "the unnecessary abrogation of responsibility of marketers and operators of AI" [10, p. 3]. This, in turn, would result in "weakening the legal protections for humans" [6, p. 275], which contrasts with law's ends. Veil-piercing, i.e., the possibility to see through the fictional personality of robots and locate the human beings who are in some way and to a certain degree responsible for their doings, is the most important point of this issue [6, pp. 286-287; 11]. Hence, every step taken towards the regulation of robotics must not interfere but enforce such possibility, especially as veil-piercing is a very delicate strategy which is therefore rarely applied [6, pp. 286-287]. As a consequence, the recourse to the 'electronic personality' legal fiction should be avoided since it entails too many legal risks. 2.3. The Pragmatist Approach Balkin's and Bryson's arguments share a common structure. Its basic premises are a functional theory of law and the determination of the ends to be attained through regulating robotics. In the epistemological context set up by such premises, the law is free to act in its best interests. This last statement implies a third premise: as Benjamin Allgrove [8, p. 71] suggests, "a clear demarcation (is) to be drawn between the philosophical and legal issues". This leaves very little doubt on how the first question addressed by this article is to be answered. Let me recall it: from a pragmatist perspective, when is it appropriate to introduce legal fictions? From this angle, the only thing that matters is how legal fictions perform within their legal context. Constructs like 'electronic personality' are both fictional and functional. They are fictional, meaning context-specific, since they are not supposed to apply outside of the social and epistemological dimension to which they belong. Moreover, they are functional, since they are introduced to serve the specific purposes of the law: they are means to given ends, that is, they are conceptual tools [6, pp. 277-280]. As a consequence, the only proper assessment of legal fictions is analogous to the kind of assessment we normally carry out to judge tools. All is needed is an evaluation of their legal usefulness [9, p. 19] or convenience [8, pp. 77-82], which implies carrying out a cost-benefit analysis. From a pragmatist point of view, the introduction of a particular legal fiction depends exclusively on such cost-benefit analyses. This kind of assessment is ultimately domain-specific. The introduction of legal fictions, then, is a technical matter which has to be discussed and settled in technical terms. Since the essentialist approach fails to acknowledge the autonomy of legal reason, it 2 Although many authors take into consideration instrumental reasons when they discuss the attribution of legal personality to artificial agents, there is no agreement on the weight to be accorded to such considerations: "where legal convenience is increased by granting legal personhood to artificial agents, this will obviously be a relevant consideration, but it should not be overstated" [2, p. 638]. loses in flexibility, elegance, and clarity. On the contrary, the superiority of the pragmatist approach lies in the economy of its functional premises, which entails a clear-cut separation between philosophical and legal concerns. 3. Legal Fictions and the Essence of Robots Within the boundaries of the pragmatist approach, things appear to be sufficiently clear. However, the premises of such approach can be partially put into question. More specifically, the separation of ontological analysis and legal reasoning may turn out to be a condition very hard to satisfy. On a theoretical level the argument is surely solid. However, is it practically feasible to keep ontological and legal considerations entirely apart from one another? To recall the second question posed in section 1.: can essentialist considerations be entirely put into brackets? 3.1. Special-Purpose Human Beings In order to elaborate on this question, let me first take into consideration Balkin's work. Even though his rebuttal of essentialism is absolute, in the course of his argument several claims on the essence of technological and robotic products are nonetheless advanced or implied. Interestingly enough, such claims are in line with his legal recommendations. The relevance of ontological conceptions is most perceivable in Balkin's discussion of the so-called "substitution effect" [1, 12]. This phenomenon builds on our well-proven tendency to anthropomorphize machines and project feelings, intentions, social skills, and (most importantly) agency onto them. As Balkin explains, we tend to treat robots as special-purpose human beings, i.e., entities which execute particular functions in ways which appear to be very similar to our own and therefore should be interpreted by reference to analogous criteria. However, this is just a false impression, which ultimately leads to the 'homunculus fallacy': "the belief that there is a little person inside the program who is making it work-it has good or bad intentions, and it makes the program do good or bad things" [12, p. 1223]. Apparently, then, the special way in which robots may be considered as human beings makes them not a particular kind of human being at all, but rather a particular kind of tool. The substitution of robots for human beings is therefore "incomplete, contextual, unstable, and, above all, opportunistic" [1, p. 57], since the essential difference that separates tools from fully autonomous and fully responsible agents cannot be actually overcome. In a word, Balkin maintains that what robotic technologies do is to mediate human actions in society: robots are tools human beings use to achieve their purposes [1, pp. 46-49] [12, pp. 1222-1223]. Framing robots by reference to the category of tool inevitably influences the discussion of how to allocate legal responsibility for harm caused by machines. Being tools, even though of a particular kind, robots should not be held directly responsible of any harm caused by their functioning. Legal responsibility should instead be allocated among the human subjects involved in their use [5, 13]: "Our interactions with robots and AI systems", Balkin writes, "are interactions with the people who are deploying these new technologies, even when we do not realize it" [1, p. 59]. Hence the need for lawmakers to focus less on robots' essence and more on the social practices new technologies mediate and on the social actors involved. In Balkin's work, then, there is indeed an ontological distinction between what it means to be human and what it means to be a robot. This distinction in fact underlies his main legal advice not to lose sight of the human factor. It is to be noted, however, that ontological analyses that challenge the categorization of robots in terms of tools also lead to different solutions concerning the issue of assigning responsibility when artificial agents are involved [14, 15]. 3.2. Purpose-Built Artifacts Balkin's expression 'special-purpose human beings' reminds strongly of Joanna Bryson's definition of artificial agents as "purpose-built artifacts" [16]. This definition is in my opinion particularly effective, since it highlights intuitively the fact that robots, no matter how much human-like or 'autonomous,' currently are and need not be more than products designed for practical use. More precisely, artificial agents are tools [17, 18] that extend "our own motivational systems [...] as a sort of mental prosthetic to our own needs and desires" [10, p. 3]. The expressions 'special-purpose human beings' and 'purpose-built artifacts' show indeed two sides of the same coin. I have already mentioned why Balkin's expression can be related to Bryson's definition: despite their apparent human-likeness, robots are just tools. On the other hand, Bryson's definition is completed by Balkin's expression in a correspondent way. In fact, the seemingly 'autonomous' behavior of artificial agents, combined with our uncertainty of what makes us really 'human,' compels us to overidentify with artificial agents and humanize them [16]. Bryson's proposition that robots should be regarded as slaves or servants [17] must also be read against this background: robots can be the kind of intelligent tools slaves were inhumanely reduced to. In sum, both Balkin and Bryson agree on one important point: while it is true that "robotics blurs the very line between people and instruments", as Calo [3, p. 515] and many others note, this line nonetheless exists and should not be overlooked [16]. Indeed, Bryson and colleagues refer to the legal side of the debate concerning the essence of artificial agents [6, pp. 283-284]. In a nutshell, this debate revolves around the question of whether robots should be framed as tools, and therefore objects of law [5, 13], or as selves, and therefore subjects of law [15, pp. 186-199; 19, 20]-with the caveat that both categories must be reconsidered. Although this would not be entirely legally binding, the authors seem willing to recognize at least some weight to such inquiries. On this point Bryson's stance is very clear: "(r)obots are not persons unless we build them to be such and then declare them to be so. Even if we do this, they will always be our creations and our responsibilities". Any attempt to "legally declare" robots "responsible for their actions [...] should in fact be considered an abdication of human responsibility for our artifacts" [17, p. 197; 10]. Bryson's ideas concerning what robots are and can do, then, are completely in line with her legal recommendations. In addition to the fact that the legal fiction of 'electronic personhood' does not perform very well within the legal system, it is also noteworthy that such legal fiction awkwardly clashes with what we know about robots' essence or essential characteristics. It goes without saying that different accounts of what it means to be an artificially intelligent robot would lead to correspondingly different legal recommendations [20]. 3.3. Summary In sum, even though the pragmatist approach is supposed to have no connection to ontological analyses, both papers touch upon ontological tenets which try as prudently as possible to set at least some very general guidelines concerning what robotic technologies are and can do. Moreover, such remarks are evidently consistent with the legal recommendations the authors submit and (most likely) influence them. It is nonetheless true that these observations are necessarily controversial, since it cannot be proven that the authors' beliefs concerning robots' essential features actually determine their regulatory advice-correlation is not causation. I can only raise the suspicion that the line that separates essentialism and pragmatism may be not as clear-cut as initially supposed. 4. 'Robotic' Legal Fictions and Social Imagination 4.1. The Social Valence of Legal Fictions These last controversial notes point to another issue that I deem most relevant to the debate on essentialism and pragmatism-that is, the social valence of legal fictions. The mutual relationship between ontological beliefs and legal concepts is problematic above and beyond the methodological perspective as well. As seen earlier, pragmatism understands legal systems as socio-cultural constructs that serve the general purpose of human environmental adaptation. As Balkin notes, this theory uncovers an interesting connection between technology and the law: "Technology", he writes, "like the law, mediates social relations between human beings, including relations of power and control" [1, p. 47]. Both technology and the law, in a sense, could be interpreted as cultural products functional to social life-that is, as socio-cultural tools. If both technology and the law can be thought of as social tools, it may be interesting to apply to legal systems as well Peter-Paul Verbeek's ideas on how technological tools impact society and human life. As he thoroughly argues in [21], tools do not simply help users to achieve particular goals: they also contribute to shaping the practical contexts in which these same tools are utilized. Technological tools, robot included [13], co-shape our perception and constitution of the world, of the entities that compose it, and of ourselves. Their influence extends beyond the context of the specific function they are built to execute. These tools have wide-ranging implications in everyday life, deeply affecting our personal, social, and cultural views of the world. Similarly, one could argue that legal systems, as socio-cultural tools, co-shape the practices they are intended to regulate and influence the social understanding of the entities involved in such practices. The legal and social understandings of robots are connected in many ways. For example, Wurah suggests that "if we welcome robots into our communities, we cannot let ourselves forget that the rights they possess will be a reflection of our society" [22, p. 69]. More generally, as Allgrove notes, "if society treats (ro)bots as autonomous actors, it puts pressure on the law to give legal effects to this social perception" [8, p. 75]. Furthermore, Calo draws attention to the fact that "the way judges conceive of robots could affect their decisions in cases involving robotics" [23, pp. 224-225]. These quotes suggest that social understandings of robots influence the legal frameworks devised to regulate them. Nevertheless, it may also be argued that the way robotic products are legally framed will feedback on the social understanding of such technologies. This, in turn, could trigger legal demands of non-fictional nature that, however, originate in social attitudes resulting in part from fictional constructs [24, pp. 1837f.]. If laws do not only reflect social attitudes and practices, but also contribute to shaping them, then the fictional character of legal fictions becomes immediately problematic in the case of robotics. The fictional character of legal fictions, a technical and subtle feature of legislation, may in fact turn out to be too easily overlooked on a social level, ultimately leading to fallacious dispositions towards robots (see the 'Android Fallacy' in [4]). Thus, in the case of robotics, the impact of legal fictions may transcend any specific regulatory domain and affect social life more in general. If this argument holds, although lawmakers may very well abstract from essential considerations when devising their strategies, we must be wary of how fictional constructs may feedback on the ways in which social actors will frame related experiences and the entities involved in them. As Lehman-Wilzig [19, p. 542] pointed out, granting legal personality to robots "does obviously mark a quantum emotional and philosophical leap from a human perspective". Legal fictions can indeed have a relevant impact on social experiences and interpretative frameworks. As a result, it is important to ask: Is legal convenience the only criterion needed to decide whether to introduce the 'electronic personality' fiction? 4.2. Robots, Anthropomorphism, and Legal Fictions It is easy to see why it is important to ask the above question. Unlike business corporations, governmental organizations, ships, idols, and other non-human entities to which legal personality has been granted, robots expose us to the many ethical, psychological, and social risks of anthropomorphism [3, pp. 547-548; 4, pp. 18-21], such as the "misassignations of responsibility and misappropriations of resources" [18, p. 63]. Several studies in human-robot interaction (HRI) show that our tendency to anthropomorphize machines is instinctual and deeply rooted in our psychology. However, as Balkin and Bryson are well aware of, overidentification with robots results in the attribution to machines of qualities which are exclusively human. This is why Bryson insists on the importance of framing robots as purpose-built artifacts or servants, and why Balkin worries about the substitution effect caused by our instinctual understanding of robots as specialpurpose human beings. Bryson and colleagues open their essay by pointing out that "[f]ictions abound with artificial human-like characters" [6, p. 274]. The recourse to legal fictions, which would in some way frame robots and human beings by reference to partially similar criteria, may feedback on social imagination and contribute to enforcing the already powerful impulse to overidentify with robots, supported as it is by popular culture and probably will be by design and commercial strategies. In the case of robots, then, it may be very difficult to prevent the fictional character of legal fictions from conditioning social imagination and informing beliefs on robots, which would in turn impact on how robots are perceived and treated in everyday life. For all the above reasons, in the case of robotics the question concerning the legal convenience of introducing fictions which imply some form of anthropomorphism may require a wider critical evaluation than the instrumental assessment presented in section 2. The focus of such a wider evaluation would not be to determine whether a fictional construct serves the ends of a regulatory strategy, but to consider how this construct would affect the general perception of robots in society [22, p. 173-174]. In sum, it would seem that the human tendency to anthropomorphize robots is a crucial element of discussion in debates concerning 'robotic' legal fictions. This tendency is independent from the law, but the law may enforce it. As both Balkin and Bryson stress, overidentification with machines is a dangerous illusion and must therefore be curbed so that its potential benefits (for instance in terms of greater acceptance of robots in society) are prudently balanced by measures that constantly point to the fictional character of this identification, protecting society from generalizing its validity. However, the mere fact that anthropomorphism must be framed as an illusion, a fiction, or a fallacy, presupposes that we know why robots cannot be considered human beings, which in turn assumes that we carry out ontological analyses of robots, and that the results of these analyses inform legislative measures as well as social initiatives. Ultimately, it seems that at least on a social level pragmatism cannot entirely dispense with some form of ontological support. 5. Conclusion This essay attempted to demonstrate that the relationship between ontological analysis and legal reasoning in regulatory approaches to robotics may be more complex than is sometimes thought. On the one hand, it is potentially very difficult to clearly distinguish between ontological tenets and pragmatic considerations. Moreover, in the specific case of the regulation of robotics, extra-legal phenomena must be taken into account as well (in particular, the many issues posed by human overidentification with robots). Because of this complexity, the impact of legal fictions has the potential to overstep the epistemological boundaries of regulatory strategies and affect social life in a wider sense. This, in turn, can modify how society conceptualizes robots in dangerous ways. In conclusion, it appears that ontological considerations around robots are not only presupposed by some pragmatist legal discussions, but also necessary to preserve the demarcation between reality and fiction in a context where the line that separates these two domains is constantly challenged. References [1] J.M. Balkin, The Path of Robotic Law, California Law Review Circuit 6 (2015), 45-60. [2] S. Chopra & L. White, "Artificial Agents-Personhood in Law and Philosophy" in: R. Lopez de Mántaras & L. Saitta (eds.), ECAI 2004, IOS Press, Amsterdam, 2004, 635-639. [3] R. Calo, Robotics and the Lessons of Cyberlaw, California Law Review 103 (2015), 513-563. [4] N. Richards & W. Smart, "How should the law think about robots?" in: R. Calo, A.M. Froomkin & I. Kerr (eds.), Robot Law, Edward Elgar Publishing, Cheltenham-Northampton, 2016, 3-22. [5] A. Bertolini, Robots as products: the case for a realistic analysis of robotic applications and liability rules, Law, Innovation and Technology 5(2) (2013), 214-247. [6] J.J. Bryson et al., Of, for, and by the people: the legal lacuna of synthetic persons, Artificial Intelligence and Law 25 (2017), 273-291. [7] U. Pagallo, The Laws of Robots. Crimes, Contracts, and Torts, Springer, Dordrecht, 2013. [8] B. Allgrove, Legal Personality for Artificial Intellects: Pragmatic Solution or Science Fiction?, SSRN (June 2004). [papers.ssrn.com/sol3/papers.cfm?abstract_id=926015 (January 31, 2018)] [9] F.M. Alexandre, The Legal Status of Artificially Intelligent Robots, [ssrn.com/abstract=926015 (January 22, 2018)] [10] J.J. Bryson, "Patiency is not a Virtue: Suggestions for Co-Constructing an Ethical Framework Including Artificial Intelligent Artifacts" in: D.J. Gunkel, J.J. Bryson, & S. Torrance (eds.), The Machine Question: AI, Ethics and Moral Responsibility. AISB/IACAP World Congress 2012, 1-5. [events.cs.bham.ac.uk/turing12/proceedings/14.pdf (January 31, 2018)] [11] S.M. Solaiman, Legal personality of robots, corporations, idols and chimpanzees: a quest for legitimacy, Artificial Intelligence and Law 25(2) (2017), 155-179. [12] J.M. Balkin, The Three Laws of Robotics in the Age of Big Data, Ohio State Law Journal 78(5) (2017), 1217-1241. [13] R. Leenes & F. Lucivero, Laws on Robots, Laws by Robots, Laws in Robots: Regulating Robot Behaviour by Design, Law, Innovation and Technology 6(2) (2014), 193-220. [14] A. Matthias, The responsibility gap: assigning responsibility for the actions of learning automata, Ethics and Information Technology 6(3) (2004), 175-183. [15] G. Hallevy, The Criminal Liability of Artificial Intelligence Entities–From Science-Fiction to Legal Social Control, Akron Intellectual Property Journal 4(2) (2010), 171-201. [16] J.J. Bryson & P.P. Kime, Just an Artifact: Why Machines are Perceived as Moral Agents, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, 2010. [www.aaai.org/ocs/index.php/IJCAI/IJCAI11/paper/view/3376/3774 (January 31, 2018.)] [17] J.J. Bryson, Building Persons is a Choice, Erwägen Wissen Ethik 20(2) (2009), 195-197. [18] J.J. Bryson, "Robots Should Be Slaves" in: Y. Wilks (ed.), Close Engagements with Artificial Companions: Key social, psychological, ethical and design issue, John Benjamins Publishing Company, Amsterdam, 2010, 63-74. [19] S. Lehman-Wilzig, Frankenstein Unbound: Towards a Legal Definition of Artificial Intelligence, Futures (December 1981), 442-457. [20] P. Čerka et al., Is it possible to grant legal personality to artificial intelligence software systems?, Computer Law and Security Review 33(5) (2017), 685-699. [21] P.-P. Verbeek, What Things Do. Philosophical Reflections on Technology, Agency, and Design, The Pennsylvania State University Press, University Park, 2015. [22] A. Wurah, We Hold These Truths To Be Self-Evident, That All Robots Are Created Equal, Journal of Futures Studies 22(2) (2017), 61-74. [23] R. Calo, Robots as Legal Metaphors, Harvard Journal of Law & Technology 30(1) (2016), 209-237. [24] E. Palmerini, Robotica e diritto: suggestioni, intersezioni, sviluppi a margine di una ricerca europea, Responsabilità civile e previdenza 6 (2016), 1816-1850.