Democratic Obligations and Technological Threats to Legitimacy Chapter 8 in Algorithms & Autonomy: The Ethics of Automated Decision Systems (Cambridge University Press, forthcoming) Alan Rubel Information School and Center for Law, Society & Justice University of Wisconsin, Madison Clinton Castro Department of Philosophy Florida International University Adam Pham Division of Humanities and Social Science California Institute of Technology This material is forthcoming in revised form in Algorithms & Autonomy: The Ethics of Automated Decision Systems by Alan Rubel, Clinton Castro, and Adam Pham (Cambridge University Press). This version is free to view and download for private research and study only. It is not for re-distribution or re-use. Please cite to the final version when available. © Alan Rubel, Clinton Castro, and Adam Pham. 2 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. 8. Democratic obligations and technological threats to legitimacy Abstract: So far in this book, we have examined algorithmic decision systems from three autonomy-based perspectives: in terms of what we owe autonomous agents (chapters 3 and 4), in terms of the conditions required for people to act autonomously (chapters 5 and 6), and in terms of the responsibilities of agents (chapter 7). In this chapter we turn to the ways in which autonomy underwrites democratic governance. Political authority, which is to say the ability of a government to exercise power, may be justifiable or not. Whether it is justified and how it can come to be justified is a question of political legitimacy. Political legitimacy is another way in which autonomy and responsibility are linked. This relationship is the basis of the current chapter, and it is important in understanding the moral salience of algorithmic systems. We will draw the connection as follows. We begin, in section 8.1, by describing two uses of technology: crime predicting technology used to drive policing practices and social media technology used to influence elections (including by Cambridge Analytica and by the Internet Research Agency). In section 8.2 we consider several views of legitimacy and argue for a hybrid version of normative legitimacy based on one recently offered by Fabienne Peter. In section 8.3 we will explain that the connection between political legitimacy and autonomy is that legitimacy is grounded in legitimating processes, which are in turn based on autonomy. Algorithmic systems-among them PredPol and the Cambridge Analytica-Facebook-Internet Research Agency amalgam-can hinder that legitimation process and conflict with democratic legitimacy, as we argue in section 8.4. We will conclude by returning to several cases that serve as through-lines to the book: Loomis, Wagner, and Houston Schools. 8.1. Two new technologies In 2011, UCLA anthropology professor Jeff Brantingham launched PredPol, which is an algorithmic system for predicting and preventing crime. The idea at the heart of PredPol is that crimes follow predictable, geographic patterns.1 Thus, it may be possible to reduce crime by 1 The analogies advocates use to explain PredPol are varied and tend to liken crime to phenomena outside of complex, modern society. One analogy is with earthquakes. After an earthquake, it is likely that there will be another one in the same area, an aftershock. Similarly for crime on this theory; certain "place-based" crimes (such as a burglary), are followed by crimes in the same area. See Erica Goode, "Sending the Police before There's a Crime," The New York Times, August 15, 2011, sec. U.S. Another analogy is to hunter-gatherers, in part because the PredPol grew out of Brantingham's work using computer models to understand hunter-gatherers. Brantingham states that "Criminals are effectively foragers ... [c]hoosing what car to steal is like choosing which animal to hunt. The same decision-making processes go into both of these choices." See Sam Hoff, "Professor Helps Develop Predictive Policing by Using Trends to Predict, Prevent Crimes," The Daily Bruin, April 26, 2013, 3 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. deploying police resources to places where predictive analytics suggests crime will occur. To facilitate this, PredPol has a graphical interface that overlays a red box (indicating a predicted crime "hotspot") on top of a city map. This allows police to concentrate in those areas in the hopes of deterring crime. There is evidence that PredPol is effective in some respects, especially in addressing property crime. And it has a degree of support. Time magazine called predictive policing2 one of the "50 Best Inventions" of 2011.3 Today, PredPol is a national leader in predictive policing technology and dozens of cities across the United States use it.4 There are, however, several important criticisms of the technology. One is that its ability to predict property crimes may not translate well to predicting (much less preventing) other crimes. Another concern is that predictive policing in general simply recreates biases in policing practices, and that predictive tools will be deployed in ways that harm communities of color, much in the same way that stop-and-frisk policies, pretrial detention, and sentencing do.5 https://web.archive.org/web/20191216105955/https://dailybruin.com/2013/04/26/professor-helps-developpredictive-policing-by-using-trends-to-predict-prevent-crimes/. We are not convinced that these analogies are helpful beyond the fact that some kinds of crime exhibit geographic patterns. 2 Note that in this chapter we will use "predictive policing" to refer to "place-based" predictive policing, predictive policing that focuses on forecasting where crimes will occur. This is often contrasted with "person-based" predictive policing. Person-based predictive policing raises similar concerns, but we will not focus on it here. For an excellent overview of these issues, see Andrew Guthrie Ferguson, The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement (New York: NYU Press, 2017), chap. 3. 3 Lev Grossman et al., "The 50 Best Inventions," Time, November 28, 2011. 4 Ferguson, The Rise of Big Data Policing, 65–67. 5 Ezekiel Edwards, "Predictive Policing Software Is More Accurate at Predicting Policing than Predicting Crime," American Civil Liberties Union (blog), accessed July 8, 2020, https://www.aclu.org/blog/criminal-lawreform/reforming-police/predictive-policing-software-more-accurate-predicting; Ferguson, The Rise of Big Data Policing, 73–74. 4 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Not long after PredPol was getting off the ground, a different set of algorithmic systems was developing in a way that would have profound implications for electoral politics. The political data analytics firm Cambridge Analytica began building specialized personal profiles and using large datasets collected from social media to stage political influence operations at scale. While use of profiling and data to influence politics is nothing new, Cambridge Analytica's actions are novel in several ways. The data on which their operation was built was massive and obtained in intrusive ways (including by gaining access via friends' permissions in Facebook). In addition, the data was collected under the imprimatur of a prestigious academic institution (Cambridge University). Finally, the kinds of information pushed to users was difficult for other to see, as it was targeted on social media. This left fewer obvious trails as it was happening than, say, television or mail ads. In parallel with the Cambridge Analytica efforts was a Russia-sponsored disinformation campaign, also making use of social media and recommendation systems. This was carried out by an organization called the Internet Research Agency (IRA). Although PredPol and Cambridge Analytica/IRA address fundamentally different parts of modern life, they both have important implications for democratic governance. PredPol is a tool that helps structure how governments exercise their enormous power to investigate and sanction crime. The ability of the state to visit harms upon its constituents is at the heart of liberal theorists' commitments to limit state power.6 This concern about state sanctioned harm is sharpened by concerns about unsanctioned state violence that is often unaccountable.7 6 Judith Shklar, "The Liberalism of Fear," in Liberalism and the Moral Life, ed. Nancy L. Rosenblum (Cambridge, MA: Harvard University Press, 1989). 7 Joanna C. Schwartz, "After Qualified Immunity," Columbia Law Review 120, no. 2 (2020): 309–88. 5 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Cambridge Analytica, in contrast, is not primarily about how a state deploys its power. Rather, it is about the connection (or disconnection) between democratic processes and governance. Put differently, it is about the necessary conditions for democratic processes to provide grounds for governments to hold power and implement policy decisions.8 Questions about justifying exercise of power via policing and questions about democratic processes and justifying power are, at root, about political legitimacy. So far in this book, we have examined algorithmic decision systems from three autonomy-based perspectives: in terms of what we owe autonomous agents (chapters 3 and 4), in terms of the conditions required for people to act autonomously (chapters 5 and 6), and in terms of the responsibilities of agents (chapter 7). Political legitimacy is another way in which autonomy and responsibility are linked. This relationship is the basis of the current chapter, and it is important in understanding the moral salience of algorithmic systems. We will draw the connection as follows. In section 8.2 we will outline a conception of political legitimacy. In section 8.3 we will explain that the connection between political legitimacy and autonomy is that legitimacy is grounded in legitimating processes, which are in turn based on autonomy. Algorithmic systems-among them PredPol and the Cambridge Analytica-Facebook amalgam-can hinder that legitimation process and conflict with democratic legitimacy, as we argue in section 8.4. We will conclude by returning to our old friends Loomis, Wagner, and Houston Schools.9 8 For the purposes of this chapter we won't draw sharp distinctions between government actions, decisions, policies, and laws. While they can be distinguished (both legally and philosophically) and different justificatory burdens may be appropriate to each, our concern here is about legitimacy and legitimation at a general level that is applicable to each. 9 For fuller description of our polestar cases see chapter 1 and see Alan Rubel, Clinton Castro, and Adam Pham "Algorithms, Agency, and Respect for Persons," Social Theory & Practice 46(3) (July 2020): 547-572; https://doi.org/10.5840/soctheorpract202062497. 6 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. 8.2. Political legitimacy Governments exercise enormous power over their denizens and expend vast resources promoting a range of policies (education, transportation infrastructure, defense, public health, commercial regulation, scientific research, information collection, parklands, safety and policing, and on and on). Those powers can be exerted in ways that are justifiable or unjustifiable, that are good or bad, and that are useful and not useful. The exercise of political authority can, in other words, be legitimate or not. In its broadest sense, political legitimacy refers to the justification of political authority, where "authority" just means having a certain political right to act coercively. But what constitutes such a right, and how could anyone ever acquire it? In this section, we discuss three conceptions of legitimacy-descriptive, democratic (or "will" based), and epistemic (or "normative authority" based)-and then offer our own, hybrid view based on a recent account from Fabienne Peter. 8.2.1. Legitimacy and descriptive criteria One family of views distinguishes legitimate from illegitimate authority in terms of purely descriptive (or empirical), as opposed to normative, criteria. This sort of view, which finds its roots in Max Weber's work,10 is often associated with the social sciences. Such descriptive accounts examine how authority (understood as people recognizing a body's ability to exert power) can be established and maintained.11 10 Max Weber, Economy and Society: An Outline of Interpretive Sociology (University of California Press, 1978). 11 This is a question of internal legitimacy, or legitimacy within a state. There may be questions as to whether a state can justify itself internationally. See Fabienne Peter, Democratic Legitimacy (New York: Routledge, 2008), chap. 1. 7 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Weber himself offered three methods of establishing and maintaining the ability to exert power: through tradition, through charisma, and through rational appeal. In other words, the continuity of traditional leadership, the transformative personal characteristics of charismatic rulers, and the inherent rationality of law can each, for better or worse, contribute to establishing and maintaining a political regime. Later theorists have refined Weber's list, distinguishing (for example) claims of traditional authority staked on "convention" from those staked on "sacredness," and distinguishing claims of charismatic authority based on "personal ties" from those based on "personal qualities."12 In the context of algorithmic systems, Ari Waldman provides an account of this descriptive sense of legitimacy as "the socially constructed propriety of authority to make decisions for others."13 Given that "[t]he managerial ethos inside corporations operating in a permissive, neoliberal regulatory environment will twist process to serve corporate ends," legitimacy requires that regulators "go beyond process to rebalance the structures of power."14 Work on this process of establishing recognition of a right to exert authority is important (and we will return to it in section 3). However, it addresses different questions than those about the normative foundation of authority. The mere fact (if it is) that people are willing to recognize a government's authority to exert power on the basis of, say, a leader's charisma, does not tell us anything at all about whether the government's actions are justifiable. To understand this, we 12 Craig Matheson, "Weber and the Classification of Forms of Legitimacy," The British Journal of Sociology 38, no. 2 (1987): 199–215. 13 Ari Ezra Waldman, "Power, Process, and Automated Decision-Making," Fordham Law Review 88, no. 2 (September 29, 2019): 614. 14 Waldman, 616. 8 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. must also understand how a right to exert authority could be justified by moral or political principles.15 8.2.2. Legitimacy and democratic criteria One normative conception of legitimacy holds that it is a function of democratic consent. Legitimacy in this sense depends on some sort of approval of authority by a state's citizens. This can be cached out in a couple of different ways. On one view, approval requires the kind of epistemic competence and non-alienation that are conditions of what we called psychological autonomy in chapter 2. Most people never agree to be governed in this way, of course, so explicit consent cannot be the requirement for legitimacy. One might then consider hypothetical agreement as the grounds for legitimacy, arguing that a sufficient condition of legitimate authority is that reasonable persons could consent to it, or one might argue that it is a necessary condition of legitimate authority that people could consent to it. Indeed, the account of psychological autonomy from John Christman that we use as an exemplar in chapter 2 links autonomy to legitimacy in this fashion. Christman writes that "the test for legitimacy of political principles should be the following: principles are legitimate only if the (reasonable) citizens to whom they apply would not be understandably alienated from them."16 What matters is not explicit agreement, but whether people would 15 For a helpful and recent overview of the topic of the grounds of political legitimacy, see Fabienne Peter, "The Grounds of Political Legitimacy," Journal of the American Philosophical Association, forthcoming. 16 John Christman, The Politics of Persons: Individual Autonomy and Socio-Historical Selves, Reissue edition (Cambridge: Cambridge University Press, 2011), 239 (emphasis in original). 9 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. identify with political principles if given adequate opportunity to reflect on them in light of their values and the sources and history of those values. Another view linking autonomy to legitimacy is based on the degree to which political bodies and decisions reflect constituent wills. What is important in legitimacy is that persons subject to authority have some say in how that authority is constituted and deployed. In this way, what Marina Oshana calls "political autonomy" more closely reflects her sense of personal autonomy than psychological autonomy.17 Legitimacy in this sense turns on the degree to which political processes afford citizens the genuine opportunity to participate, rather than principles reflecting their wills. Both views of autonomy ground legitimacy in democratic will. Rawls, for instance, offers a principle of legitimacy that is defined in terms of "public reason." On this view, political power is justified "when it is exercised in accordance with a constitution the essentials of which all citizens as free and equal may reasonably be expected to endorse in the light of principles and ideals acceptable to their common human reason."18 Similarly, Pettit's account is grounded in "popular control." He argues that political power is justified when all citizens have equal access to the prevailing system of influence over the government and that system imposes a direction welcomed by all.19 In both cases, the account of legitimacy is premised on the notion of individuals as the "self-originating sources of valid claims," in the sense that "their claims have 17 Marina Oshana, Personal Autonomy in Society, 1st ed. (Aldershot, Hants, England ; Burlington, VT: Routledge, 2006), 97–100. 18 John Rawls, Political Liberalism (New York: Columbia University Press, 2005), 137. 19 Philip Pettit, Just Freedom: A Moral Compass for a Complex World, 1st edition (New York: W. W. Norton & Company, 2014), chap. 5. 10 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. weight apart from being derived from duties or obligations specified by the political conception of justice, for example, from duties and obligations owed to society."20 What gives their claims this sort of weight is their autonomous wills. Both Rawls's and Pettit's views align with descriptive accounts in that both Rawls and Pettit acknowledge that legitimacy is tied to stability. However, they diverge from descriptive accounts in maintaining that legitimacy requires that political authority meet normative criteria as well. Specifically, political authority must obey certain rules and democratic procedures. Rawls describes the ideal of political legitimacy not in terms of stability per se (that is, in terms of a "modus vivendi"), but in terms of "stability for the right reasons," where each citizen is compelled to obey the public constitution from their own individual perspective rather than merely out of self-interest. And Pettit describes the ideal of legitimacy in terms of passing a "tough luck" test, according to which authority is legitimate when people can think that "when public structures and policies and decisions frustrate their personal preferences, that is just tough luck."21 Note that these normative criteria for political legitimacy are versions of the reasonable endorsement principles we outline in chapter 3. Since these principles index the degree of legitimacy to facts about the wills of constituents, we can, following Fabienne Peter, call them "will" principles of legitimacy. 8.2.3. Legitimacy and epistemic criteria A third view of legitimacy ties it to whether systems or decisions meet epistemic criteria. 20 John Rawls, "Justice as Fairness: Political Not Metaphysical," Philosophy & Public Affairs 14, no. 3 (1985): 242. 21 Pettit, Just Freedom, 112. 11 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. To understand this set of views, it is useful to start with what Peter calls the "normative facts" view. On this conception, what matters for legitimacy turns on whether political decisions track the truth of the matter as to what is morally right to do. That is, the normative facts view assumes something akin to moral realism, thinks that propositions about what governments should do can be true or false, and pegs legitimacy to whether such decisions indeed track those normative facts. And they do so regardless of what people's beliefs about them are. The problem of a normative facts view is that it leaves out the link between legitimacy and autonomy. That is, legitimate decisions need not be ratified in any sense by the people subject to them. Peter explains this as a violation of what she calls the "access constraint": political legitimacy, whatever it is, must "be such that it can settle political deliberation (at least temporarily)," and to do this, it "must involve the attitudes of at least some citizens." Therefore, if normative facts are relevant to our political decision-making, "it will be through our beliefs about them." In this vein (i.e., bridging normative facts and beliefs about normative facts), Raz offers a "service" conception of legitimacy. On this view political authority is legitimate only if it is made in service of people's underlying reasons. The "normal" justification for authority, in other words, is that authority done right does nothing more than guide people according to the reasons they already have. As Raz puts it, "the normal way to establish that a person has authority over another person involves showing that the alleged subject is likely better to comply with reasons which apply to him [...] if he accepts the directives of the alleged authority as authoritatively binding and tries to follow them, rather than by trying to follow the reasons which apply to him 12 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. directly."22 Later theorists following Raz have explained further how this sort of epistemic preemption might work. David Enoch, for instance, extends Raz's notion of a duty as a special sort of reason, describing authority as giving rise to obligation when there is some "normative structure in the background" allowing the authority to give its subject those sorts of reasons.23 On wildlife tours, the tour guide often serves (and expects to serve) as this sort of preemptive decision-maker. The guide is the authority on, for instance, when the group needs to leave a group of animals behind rather than continue to take pictures-because of the dangerous nature of the circumstances and the special knowledge the guide has acquired through experience over time. In any case, when the normal justification is present, Raz argues, the duties prescribed by the authority can override even the dictates of the democratic will, and thus can offer grounds for "a ready embrace of various paternalistic measures" which would not pass democratic muster.24 Here, we might find the justification for such policies as smoking bans, bans on trans fats, helmet laws, and mask mandates. 8.2.4. The disjunctive conception of legitimacy As Peter points out, neither the democratic nor the epistemic criteria seem on their own to fully encapsulate the normative grounds of political legitimacy.25 If we accept what she calls a "willbased" conception of political legitimacy, exemplified by Rawls's and Pettit's democratic 22 Joseph Raz, The Morality of Freedom (Oxford: Oxford University Press, 1988), 53. 23 David Enoch, "Authority and Reason-Giving," Philosophy and Phenomenological Research 89, no. 2 (2014): 31. 24 Raz, The Morality of Freedom, 422. 25 Peter, "The Grounds of Political Legitimacy." 13 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. criteria, we run into what Peter calls the arbitrariness objection: that "[t]he validity of at least some practical claims depends on third-personal sources of validity," and thus, that these willbased conceptions "support undue arbitrariness in political decision-making." We cannot hold the wills of all citizens to be politically relevant; individuals are not self-originating sources of valid claims but are instead often deeply irrational or unreasonable. Hence, democratic processes (via voting, consensus, public reason, or the like) are not sufficient on their own to justify exercise of political authority. There are, in other words, exercises of authority that are illegitimate even if they genuinely reflect citizen will. They may be so ill-conceived and so harmful, or they may be so detrimental to democratic processes and structures themselves, that they can't be legitimated by democratic processes. A purely "belief-based" conception of political legitimacy (corresponding to Raz's and Enoch's epistemic criteria) is also inadequate. Peters argues that these accounts run the problem of "epistemic underdetermination." Specifically, there are few (if any) political decisions for which there is sufficient evidence to ensure that they are optimal. As Peter puts it, "the epistemic circumstances of politics are such that for most political decisions, there will not be a decisive normative authority."26 So even if we reject democratic adjudication as the univocal source of normative authority, the epistemic circumstances are often so complex and uncertain that normative authority remains difficult or impossible to establish. The main tension is that persons' wills and epistemic criteria both seem to be key conditions of justifiable exercise of political 26 For a longer discussion of this problem, see Fabienne Peter, "The Epistemic Circumstances of Democracy," in The Epistemic Life of Groups: Essays in the Epistemology of Collectives, ed. Miranda Fricker and Michael Brady (Oxford: Oxford University Press, 2016), 133–149. 14 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. authority. Thus, the determinative question seems to be how to incorporate the autonomous wills and beliefs of citizens, given the need for policy that is appropriately responsive to facts.27 To address to these issues, Peter defends a hybrid account of the normative grounds of political legitimacy. This account holds that legitimacy includes "both responsiveness to normative authority and adjudication between valid, but conflicting, claims as grounds of legitimacy." The conception itself is "disjunctive," which is to say that legitimacy can derive from either will or belief. Moreover, it favors normative authority over democratic processes; legitimacy is tied to "how the decision reflects normative authority, when normative authority can be established, or, when normative authority cannot be established, [...] how it adjudicates between the conflicting, but valid claims made by the citizens." In other words, democratic adjudication is secondary; it is a source of legitimacy only "as a response to difficulties with establishing normative authority." Peter's hybrid account of the grounds of legitimacy is on the right track, but we want to refine it slightly. We cannot define our way out of the threats to legitimacy presented by our most difficult and contentious cases simply through disjunctive addition. It is not as if the arbitrariness problem somehow disappears in cases where normative authority cannot be established. (Indeed, it seems more likely that the two problems are self-reinforcing and thus, run together.) It may well be possible to balance power among parties to a conflict in some cases where normative authority is epistemically underdetermined, but we should not mistake this modus vivendi for an arrangement that is genuinely politically legitimate. 27 David M. Estlund, Democratic Authority: A Philosophical Framework (Princeton, N.J.: Princeton University Press, 2008). 15 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Peter's concern with securing a secondary ground of legitimacy beyond normative authority is connected to her concern with the function of political legitimacy. She writes that "[a] plausible conception of political legitimacy should be able to settle which decisions are and which are not legitimate even under unfavourable epistemic circumstances," and this is something that can often only be settled by the balancing of wills. Nor, she argues, can we "claim that all decisions made in the absence of decisive normative authority would be illegitimate," because this "would itself amount to a political decision about which we could ask whether it is legitimate," leading to a regress. Peter considers the secondary role of democratic adjudication in the context of a wellentrenched topic of disagreement: abortion. If we grant for the sake of argument that there is no decisive normative authority on that issue, it is not clear that any political method of adjudicating the conflicting claims on the issue will ever be able to provide the sort of consolation that might soothe parties who have been disenfranchised by "tough luck," to say nothing of providing stability for the right reasons. This seems like precisely the sort of issue that democratic adjudication can settle, but not in the "right" way. To see the grounds and function of political legitimacy correctly, we need to keep the secondary notion of democratic adjudication separate from the primary notion of normative authority. This is because there are political decisions, systems, or regimes that might be democratically ratified, but that are illegitimate in the epistemic, normative authority sense. Thus, there will be some policies that appear legitimate in the democratic sense, but are not legitimate in the normative authority sense. Our view is similar to Peter's hybrid model, with the caveat that policies can be legitimate via a combination of normative authority and democratic 16 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. will. As evidence of good policy increases (that is, as it becomes increasingly evident that some policy, law, or action is justified via normative authority), then the stringency of the democratic facet of legitimacy is weaker. 8.3. Legitimating processes As we explain in the previous section, there are two facets to normative legitimacy. One, characterized by "will" theories generally, is vulnerable to arbitrariness. The other is characterized by normative authority, which is in turn a function of justified belief in structures, policies, and actions of states that are in fact good. This second facet isn't vulnerable to arbitrariness in the way that will theories are. That is because justification is baked into the definition. Any policy, structure, or action that is both correct and justifiably believed to be correct is, ipso facto, legitimate. The primary limitation of the normative authority facet of legitimacy is epistemic; determining what is in fact the right policy or action is difficult, to say the least (cf., section 1.5). When we consider technological threats to legitimacy-gaming of algorithms in social media, use of algorithms in predictive policing, for example-it is tempting to move straight from a conception of legitimacy to a claim that some set of institutions, policies, or structures are themselves illegitimate, all things considered. It will no doubt be true that some political actions will fail to meet both the will conditions and the normative authority conditions for legitimacy and will, hence, be unjustifiable exercises of authority. However, for any particular policy taking place within a complex set of social and political structures against background facts that are themselves challenging to interpret, all-things-considered judgments about legitimacy will be difficult. Hence, it is a far larger task than we can address here. Moreover, making that kind of 17 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. final judgment about legitimacy itself would take us too far afield from the subject of this book, which is about autonomy. Finally, legitimacy (like autonomy) is not a binary, success concept. It is not something that an institution, polity, law, or action simply has or lacks. There is a spectrum of legitimacy, and limitations based on will or based on normative authority will often diminish legitimacy without warranting a conclusion that a policy decision crosses some floor below which it is illegitimate. For these reasons, our focus here is on legitimation.28 Will theories, normative authority theories, Peter's disjunctive theory, and our hybrid interpretation of Peter's view all rely on exercises of persons' autonomy as a foundation for legitimacy of governments, laws, policies, and the like. That process of exercising autonomy, either through will or through justified beliefs about normative claims (or both), is the mechanism by which use of political authority is justified. And in turn, members of a polity have the responsibility to use their autonomy to understand and support (or not support) actions by their government.29 It is that process of legitimation that algorithmic systems can disrupt. In other words, algorithmic systems may in some cases hinder people from fulfilling their responsibilities of legitimating government policies, actions, laws, and the like. 28 Note that "legitimation" in this normative sense is different from, though related to, the descriptive sense of legitimation in descriptive accounts of legitimacy. The descriptive sense of legitimation refers to the processes by which a governing entity establishes and maintains power, regardless of whether those process justify that power. Descriptive legitimation may be necessary for normatively legitimate policies to function. 29 To be clear, it is neither possible nor desirable that every member of a polity understand every kind of government institution, law, and action. Rather, there are some broad states of affairs that most any member of society has a responsibility to understand and support (or not): some conception of fair representation and enfranchisement, basic human rights. Others will have responsibilities based on their communities, experiences, and expertise. The responsibility is to exercise autonomy (properly understood as social and relational) in some facets of social life. 18 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Our argument for the centrality of autonomy to legitimacy begins with the conception of legitimacy outlined above. For political authority to be justified, it must be based on either democratic will or normative authority (i.e., where the exercise of political authority is good and there is justified belief that it is good), or a degree of both. As normative authority increases- that is, where policy is good and there is excellent epistemic justification for it-the degree to which it must be supported by mere will decreases. Both paths to legitimate authority are based on a process of legitimation, each of which is itself grounded in autonomy. We explained in section 8.2.2 that will-based theories are based on some version of agreement, public reason, or hypothetical consent, and that Christman and Oshana explicitly link legitimacy to their conceptions of autonomy. The idea is straightforward. For one's agreement to a system of governance and to the actions, policies, and decisions of a government to carry force, the agreement must be based on conditions that meet the criteria of autonomy and freedom. One must be epistemically competent, the reasoning upon which one's agreement is based must satisfy authenticity conditions (as Christman puts, one must not be understandably alienated from them). One should have the room to determine whether to support political decisions by having a degree of substantive independence. Moreover, people performing their legitimating responsibilities should be free of substantial impediments to the quality of their agency. As we explain in chapter 5, one's agency may be impaired by affective, deliberative, and social challenges. Where the quality of their agency is sufficiently challenged in those ways, their ability to fulfill the responsibility of legitimating government policies is limited. 19 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. The link between autonomy and legitimation by normative authority is not as intuitively clear as between autonomy and will theories. Normative authority requires, first, that a policy, law, or action by a government be a good one. But the mere fact that a policy is good cannot confer legitimacy because of what Peter calls the "access constraint." Legitimacy demands that people believe that the policy is good, and that they do so for sound reasons. Put another way, the legitimation process requires that people have sufficient information and understanding to assess a policy and believe that it is justified on its merits. The ability to do that is based on autonomy. To form justified beliefs about whether a policy is good, one must be epistemically competent, they must have some sense of value from which they are not alienated, and their reasoning must be substantively independent enough that their ability to reason is not compromised. Finally, the quality of their agency should not be so diminished that it undermines the degree to which their beliefs and actions are indeed their own. 8.4. Technological threats to legitimacy Before we apply our arguments about legitimacy, legitimation, and autonomy to algorithmic systems, let's summarize the conception of legitimacy that we have developed. 1. For a government policy, law, or action to normatively politically legitimate, requires either (a) normative authority, (b) democratic will (within certain bounds, dictated by (a)), or (c) some combination of both (a) and (b). 20 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. 2. For a policy, law, or action to be legitimate via normative authority requires that the policy, law, or action in fact be good, and meet the access constraint. 3. The access constraint functions to ensure that autonomy enters the process by having belief-forming mechanisms in place under sufficient quality of agency. 4. For a policy, law, or action to be legitimate via democratic will requires some process of voting or public reason. 5. For a process of voting or public reason to confer legitimacy requires that persons engaged in that process to do so autonomously. With that in mind, let's return to PredPol, Cambridge Analytica, and the IRA. 8.4.1. PredPol, normative authority, and legitimation Our analysis of PredPol and legitimation is centered on legitimacy through normative authority. Recall that legitimacy via normative authority involves both normative claims (i.e., that some policy is in fact a good one) and the access constraint. The access constraint requires that the reason some policy is implemented is that people implementing it have epistemically sound reason to believe that the policy is in fact good. Based on these criteria, there are four key questions for determining whether a policy is legitimate based on the normative authority view: (1) is the policy a good one? (2) Is there ample evidence that the policy is a good one? (3) Does 21 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. the evidence that the policy is good drive the beliefs about the policy? And (4) do the beliefs about the policy reflect the actual implementation of the policy? We will consider the first two questions together, since we cannot address the question of whether the policy is good independent of evidence for it. So, is use of PredPol in fact good policy? Certainly, reduction in crime is a worthwhile objective. Hence, whether use of PredPol is good policy will turn on whether it is effective, whether it has bad consequences that offset any good ones, or whether it impinges other claims. There is some evidence PredPol is effective. In a July 2011 pilot test, the Santa Cruz, CA police department piloted a crime-deterrence program using PredPol. Zach Friend, the department's crime analyst, reported encouraging results. Burglaries in 2011 were down 27 percent that July, compared with July in the previous year. Speaking to the question of effectiveness, Friend added, "The worst-case scenario is that it doesn't work and we're no worse off," (though who the "we" refers to is unclear).30 However, the evidence of PredPol's efficacy is equivocal. Santa Cruz's drop in burglaries in 2011 is not enough to conclude that PredPol was the cause; after all, cities are complex entities and a one-year drop in one category of crime does not warrant a conclusion about a single cause and tells us nothing about other crimes. There have been only a few academic studies on predictive systems like PredPol. Those that exist yield mixed results, with some showing predictive methods having some crime-reducing effects, but others showing it has no effect at 30 Goode, "Sending the Police before There's a Crime." 22 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. all.31 For example, a study by the RAND Corporation testing a place-based system like PredPol found that the predictive system had no statistically significant effect on crime overall.32 There is also reason to think that use of PredPol has negative effects that offset whatever advantages in crime reduction it confers. Several cities using predictive policing technology have found that police simply stay in the locations indicated by the algorithm. That is, they stayed in the "red box" that PredPol overlays on a map. Susan Merritt, San Francisco Police Department's chief information officer, remarked that "In L.A. I heard that many officers were only patrolling the red boxes, not other areas [...] People became too focused on the boxes, and they had to come up with a slogan, 'Think outside the box'."33 And there is evidence that this has deleterious effects. Patrol officers stayed in the "red boxed" areas and engaged in "intelligence gathering," for example questioning people, running people's records, investigating for potential (though unreported) narcotics use, and so forth. This, in turn, had the effect of slowing response times to emergencies.34 A further consequence that offsets whatever benefits PredPol has is disproportionate use. The Los Angeles Police Department (LAPD) has used predictive methods as a reason to deploy helicopters (as opposed to patrol cars) to the 500 foot by 500 foot boxes that denote "hot spots." 31 Ferguson, The Rise of Big Data Policing, 72; Albert Meijer and Martijn Wessels, "Predictive Policing: Review of Benefits and Drawbacks," International Journal of Public Administration 42, no. 12 (September 10, 2019): 1031– 39. 32 Priscilla Hunt, Jessica Saunders, and John Hollywood, "Evaluation of the Shreveport Predictive Policing Experiment" (Santa Monica, CA: RAND Corporation, 2014), https://www.rand.org/pubs/research_reports/RR531.html. 33 Darwin Bond-Graham, "All Tomorrow's Crimes: The Future of Policing Looks a Lot like Good Branding," SF Weekly, October 30, 2013, https://www.sfweekly.com/news/all-tomorrows-crimes-the-future-of-policing-looks-alot-like-good-branding/; Ferguson, The Rise of Big Data Policing, 79. 34 Ferguson, The Rise of Big Data Policing, 79; Hunt, Saunders, and Hollywood, "Evaluation of the Shreveport Predictive Policing Experiment." 23 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. These practices outrun the design the predictive tool for several reasons. For one, it is difficult to discern unspecified criminal activity from a helicopter. And helicopters' deterrent effect is fleeting. Criminologist Geoffrey Alpert-a specialist in the use of helicopters in policing-says that helicopters, which are loud and obvious, will at best deter criminals while they are overhead, but that criminals will simply return when the helicopters are not around.35 Moreover, accommodating helicopter flight paths, the search area had to be expanded well beyond the "red box," conflicting with the purported specificity of PredPol.36 Add to this the annoyance residents experience having loud helicopters overhead, and there are several negative consequences offsetting PredPol's advantages. Finally, there are potential rights impingements relevant in determining whether PredPol is good policy. Predictive models indicating a "high crime area" can be a factor in determining whether police have legal authority to stop and frisk a person.37 Hence, people may be stopped and frisked more often in a PredPol-designated area, and such stops are more likely to be permissible under the Fourth Amendment. However, it is another question whether merely being in a high crime area can help morally justify a stop. And because of the source of data, the patterns of high-crime areas are skewed towards neighborhoods in which residents are predominantly from racial and ethnic minorities. 35 Kate Mather and Richard Winton, "LAPD Uses Its Helicopters to Stop Crimes before They Start," Los Angeles Times, March 7, 2015, sec. California. 36 Ferguson, The Rise of Big Data Policing, 81; Mather and Winton, "LAPD Uses Its Helicopters to Stop Crimes before They Start." 37 Ferguson, The Rise of Big Data Policing. For example, in Illinois v. Wardlow, 528 U.S. 119 (2000) the Supreme determined that simply being in high crime area did not suffice to justify a Terry stop, but it can be relevant in determining whether other facts are sufficiently suspicious for a stop. 24 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. So, even if use of PredPol has some positive effects on a justifiable policy goal, it is not warranted to conclude that it is good policy overall and that there is ample evidence for it being good policy. Let's assume that the answer to the third question-whether evidence that the policy is good drives beliefs about the policy-is yes. That is not to say that the inference is correct; the access constraint merely requires a connection between evidence and belief on the assumption that the policy is good and justifiable. The legitimation problem for use of PredPol comes in steps (3) and (4). It is possible that what evidence there is that PredPol can help reduce some types of crime grounds municipalities' use of the technology. However, the actual implementation of PredPol does not appear to reflect the evidence of its efficacy. Even if PredPol is well-suited for predicting some kinds of crimes, the evidence for that does not warrant belief about PredPol's use as a general law enforcement tool, much less as a reason to park police officers to sit in red-boxed areas or to act as general investigators (and slow responses to emergencies). It likewise does not support belief that deploying helicopters to fly over red-boxed and surrounding areas is warranted. It also does not provide grounds for thinking that individual persons who happen to be in red-boxed areas are themselves engaged in criminal activity. To reiterate, the access constraint is the requirement that the fact (if it is a fact) that a policy is a good one be the ground for belief in the policy's justification. Here, the fact the PredPol is effective in one area instead grounds the apparent belief that it is a good tool for intensive policing of discrete areas and conducting stops of individuals. In this way, the evidence for PredPol's efficacy in one area actually circumvents an important legitimating process. The evidence for its efficacy in one area stands in for evidence of efficacy for the tasks in which it is actually used. This is a barrier for citizens in being able to form beliefs about whether a policy is a good one. In other words, deploying PredPol well 25 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. outside the range of circumstances for which there is evidence of its efficacy undermines citizens' abilities to fulfill their responsibility of exercising their autonomy in legitimating policy. Here we want to very clear about the limitations of this discussion. The mere fact that a policy is not good does not render it illegitimate. Any kind of policymaking (broadly construed) is complex, takes place under epistemic limitations, and will affect lots of people with considerable interests in the policy. Moreover, for a policy to be "good" in the strong sense necessary for it to be legitimate via normative authority is a very tall order. And the fact that use of PredPol does not seem to clear that threshold is therefore unsurprising. Use of PredPol might instead be legitimate based on the democratic, will-based view. Hence, our argument is emphatically not an attempt to make an all-things-considered judgment that using PredPol is not democratically legitimate (perhaps it is, perhaps it isn't). Rather, as we emphasize above, our purpose here is to examine autonomy and the process of legitimation. And on that question, PredPol is an informative case because it precludes citizens' abilities to legitimate predictive policing by undercutting their ability assess its policy rationale. 8.4.2. Cambridge Analytica, the IRA, and will-based authority What are often referred to as the "Cambridge Analytica scandal" and "Russian hacking case" are in fact an interconnected, overlapping, and confusing welter of activities, organizations, tools, and technologies that grew up before and became widely known in the wake of the 2016 UK Brexit campaign and the 2016 U.S. presidential election. The events are well-documented in that 26 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. there have been multiple tell-all books,38 government investigations,39 academic reports,40 and news stories.41 However, the interwoven stories are obscured by their very complexity, redactions in official reports and indictments, disinformation and deflection campaigns, and failures of public institutions to follow through with further investigations. Given the scope of the stories, we cannot possibly do them justice here. However, we can pull out a few important threads to ground our arguments about legitimation. Cambridge Analytica is a data analytics and political consultancy firm based in London, UK. It is an offshoot of the SCL Group. In the years before the Brexit campaign and the 2016 U.S. presidential election, the firm became interested in using the gigantic, rich datasets generated on social media platforms-particularly Facebook-to build ever more sophisticated models of voters and behavior. In order to pursue this work, they partnered with Cambridge University psychologist Aleksandr Kogan, who was already collecting social media information for his own research. Kogan had developed (along with other researchers) a personality app called "This Is Your Digital Life." The app, which included a "lengthy psychological 38 Christopher Wylie, Mindf*ck: Cambridge Analytica and the Plot to Break America (New York: Random House, 2019); Brittany Kaiser, Targeted: The Cambridge Analytica Whistleblower's inside Story of How Big Data, Trump, and Facebook Broke Democracy and How It Can Happen Again (Harper, 2019). 39 U.S. Department of Justice, "Report on the Investigation into Russian Interference in the 2016 Presidential Election, Volume I ('Mueller Report')" (Washington, D.C.: U.S. Department of Justice, March 2019); Select Committee on Intelligence, United States Senate, "Report of the Select Committee on Intelligence, United States Senate, on Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume I: Russian Efforts against Election Infrastructure with Additional Views" (Washington, D.C: Select Committee on Intelligence, United States Senate, 2019). 40 Philip N. Howard et al., "Social Media, News and Political Information during the US Election: Was Polarizing Content Concentrated in Swing States?," Oxford Internet Institute, Project on Computational Propaganda, 2017; Renee DiResta et al., "The Tactics & Tropes of the Internet Research Agency," U.S. Senate Documents, October 1, 2019. 41 Carole Cadwalladr and Emma Graham-Harrison, "Revealed: 50 Million Facebook Profiles Harvested for Cambridge Analytica in Major Data Breach," The Guardian, March 17, 2018, sec. News; Cecilia Kang and Sheera Frenkel, "Facebook Says Cambridge Analytica Harvested Data of up to 87 Million Users," The New York Times, April 4, 2018, sec. Technology. 27 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. questionnaire,"42 was actually only downloaded by around 270,000 users, but it ultimately harvested user data from those users and from their friends, a group which might number up to 87 million.43 This data was shared with Cambridge Analytica. On the basis of psychological profiles and data about Facebook users, Cambridge Analytica offered micro-targeted advertising based on their data-driven psychometrics. They did this first for Republican party nomination seeker Ted Cruz.44 When Cruz dropped out of the nomination race, they shifted their operation to candidate Donald Trump. The targeted advertising and social media campaign spearheaded by Cambridge Analytica is of particular note. Christopher Wylie is a data scientist and former employee of Cambridge Analytica who eventually became a whistleblower and author of a book about the company. He describes several key features of the company's strategies. For example, he describes a project targeting African American voters with messages designed to decrease their motivation for voting. He also describes using the techniques of using its influence to create inperson meetings of angry citizens.45 As Wylie put things, in the original report published by The Guardian: "We exploited Facebook to harvest millions of people's profiles. And built models to 42 Nicholas Confessore, "Cambridge Analytica and Facebook: The Scandal and the Fallout so Far," The New York Times, April 4, 2018, sec. U.S. 43 Kang and Frenkel, "Facebook Says Cambridge Analytica Harvested Data of up to 87 Million Users." 44 Harry Davies, "Ted Cruz Campaign Using Firm That Harvested Data on Millions of Unwitting Facebook Users," The Guardian, December 11, 2015, sec. US news, https://www.theguardian.com/us-news/2015/dec/11/senator-tedcruz-president-campaign-facebook-user-data. 45 Wylie, Mindf*ck: Cambridge Analytica and the Plot to Break America, 127–29. 28 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. exploit what we knew about them and target their inner demons. That was the basis the entire company was built on."46 The Russian Internet Research Agency (IRA) is a Russian state-supported organization that was established in 2013. It operates "like a sophisticated marketing agency," has trained "over a thousand people to engage in round-the-clock influence operations," and has targeted citizens in variety of places, including Russia, Ukraine, and the United States.47 In February 2018, the U.S. Department of Justice indicted the IRA and a number of Russian nationals who worked with the IRA for conspiracy to commit fraud, wire fraud, and bank fraud.48 The charges were based in part on disinformation "active measures" carried out on social media.49 These measures included creation of fictitious groups that engaged social media users with inflammatory and divisive content. The IRA created fake profiles that followed other IRA sites and groups, thereby increasing their footprint and visibility to real people on social media platforms. To reach larger audiences, the IRA purchased Facebook ads that were then placed in potential followers' Facebook newsfeeds.50 The inflammatory, divisive content promoted by the IRA sought to engage with social media from a range of political affiliations. The U.S. Senate Select Committee on Intelligence found that the IRA's targeting was particularly intensive toward African Americans. It writes, "By far, race and related issues were the preferred target of 46 Cadwalladr and Graham-Harrison, "Revealed: 50 Million Facebook Profiles Harvested for Cambridge Analytica in Major Data Breach." 47 DiResta et al., "The Tactics & Tropes of the Internet Research Agency," 6. 48 U.S. v. Internet Research Agency, LLC, No. 1:18-cr-00032 (U.S. District Court for the District of DC February 16, 2018). 49 U.S. v. Internet Research Agency, LLC. 50 U.S. Department of Justice, "Report on the Investigation into Russian Interference in the 2016 Presidential Election, Volume I ('Mueller Report')," 25. 29 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. the information warfare campaign designed to divide the country in 2016."51 It did this by creating pages and groups that sought to engage users that the social media platform identified as African American and by targeting geographic areas that are predominantly African American. These include pages such as "Blacktivist" and social media posts related to police shootings and NFL player kneeling protests of police shootings.52 Other groups the IRA targeted with incendiary messaging include groups with names that imply a range of political dispositions ("Being Patriotic," "Stop All Immigrants," "Secured Borders," "Tea Party News," "Black Matters," "Don't Shoot Us," "LGBT United," and "United Muslims of America."53 The IRA's social media accounts "reached tens of millions of U.S. persons" and had "hundreds of thousands of followers."54 These accounts were used to organize in-person rallies, to recruit activists to perform political tasks and promote IRA content.55 The IRA's activity is linked to social media users being exposed to a great deal of misinformation. A study of Twitter by the Oxford Internet Institute found that in the run up to the 2016 presidential election, "users got more misinformation; polarizing, and conspiratorial content than professionally produced news."56 51 Select Committee on Intelligence, United States Senate, "Report of the Select Committee on Intelligence, United States Senate, on Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume II: Russia's Use of Social Media and Additional Views" (Washington, D.C: Select Committee on Intelligence, United States Senate, 2019), 6. 52 Select Committee on Intelligence, United States Senate, 6–7. 53 U.S. Department of Justice, "Report on the Investigation into Russian Interference in the 2016 Presidential Election, Volume I ('Mueller Report')," 24–25. 54 U.S. Department of Justice, 26. 55 U.S. Department of Justice, 31–32. 56 Howard et al., "Social Media, News and Political Information during the US Election: Was Polarizing Content Concentrated in Swing States?," 1. 30 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. The breadth of activities by the IRA is striking. Its activities range across social media platforms, including Facebook, YouTube, Instagram, and Twitter. It targets groups from a range of social and political perspectives. Its tactics include aiming to suppress votes, aiming to direct voters toward third parties, aiming to depress turnout, encouraging secessionist movements (including in Texas and in California, comparing them to Brexit), and denigrating a range of leaders across parties.57 They also aimed to both seed news media with content generated by Russian disinformation campaigns and to undermine trust in traditional news media.58 There is no official recognition or definitive documentation connecting Cambridge Analytica and Russian active measures. In his book, Wylie recounts Cambridge Analytica's involvement with Russian actors, message testing about Vladimir Putin, close connections to people involved pro-Russia factions in Ukraine politics,59 and involvement with Russian oil firm Lukoil.60 SCL's promotional materials indicate that they did work for Russia. However, SCL denies being involved in political campaigns at behest of Russia, insisting that their work in Russia was for "private" interests.61 It is worth noting here that the distinction between state and nominally private interests in kleptocracy is blurry. So, the existence and nature of the connection between the IRA and Cambridge Analytica are unclear. However, what matters here is not whether there was a single group of people acting 57 DiResta et al., "The Tactics & Tropes of the Internet Research Agency," 8–10. 58 DiResta et al., 65–66. 59 Wylie, Mindf*ck: Cambridge Analytica and the Plot to Break America, 142. 60 Wylie, 141, 155–56. 61 Danny Hakim and Matthew Rosenberg, "Data Firm Tied to Trump Campaign Talked Business with Russians," The New York Times, March 17, 2018, sec. U.S.; Wylie, Mindf*ck: Cambridge Analytica and the Plot to Break America, 139–59. 31 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. in concert to sow anger and seed disinformation in the U.S. and elsewhere. Rather, our focus here is on the nexus of intrusive data collection (itself possible based on Facebook's weak privacy protections and permissive terms of service (themselves difficult to comprehend and rarely read), and the imprimatur of Cambridge University), predictive analytics using that data to better target influence, and algorithmic systems that suggest advertising to clients (see section 7.4) and promote content to users. That is, our concern here is a socio-technical, big-dataand algorithmically aided group of systems that affect the attitudes, beliefs, dispositions, and actions of people within democratic states. Whether Cambridge Analytica entered into agreements or acted in explicit concert with the IRA is neither here nor there for our purposes in this chapter. We should also emphasize that the effect of Cambridge Analytica's and the IRA's actions on the outcomes of particular elections is unclear, and no single set of events could ever be said to be the sole cause of complex phenomena like election outcomes and broad policy-making. It is disputed just how well these kinds of tactics actually work. Kogan argued that the "accuracy" of the data was exaggerated in media reports and that "[w]hat Cambridge has tried to sell is magic."62 At first, the Cruz campaign credited the data-driven approach with its win in the 2016 Iowa caucus,63 but NPR reports that, as the election cycle moved forward, the campaign grew more skeptical and eventually phased out the psychological profiling after later losing the South Carolina primary.64 62 Matthew Weaver, "Facebook Scandal: I Am Being Used as Scapegoat – Academic Who Mined Data," The Guardian, March 21, 2018, sec. UK news. 63 Tom Hamburger, "Cruz Campaign Credits Psychological Data and Analytics for Its Rising Success," Washington Post, December 13, 2015, sec. Politics. 64 Scott Detrow, "What Did Cambridge Analytica Do during the 2016 Election?," NPR, accessed June 17, 2020, https://www.npr.org/2018/03/20/595338116/what-did-cambridge-analytica-do-during-the-2016-election. 32 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Nonetheless, the tactics are important from the standpoint of political legitimacy. As a team of psychologists have recently shown, there is an increasing amount of evidence for "the effectiveness of psychological targeting in the context of real-life digital mass persuasion," and "tailoring persuasive appeals to the psychological profiles of large groups of people allowed us to influence their actual behaviors and choices."65 The Cambridge Analytica case, they comment, "illustrates clearly how psychological mass persuasion could be abused to manipulate people to behave in ways that are neither in their best interest nor in the best interest of society."66 This creates a problem of legitimacy. Whereas in the PredPol case we consider the normative authority path to legitimacy, here our concern is the will, or democratic path. Recall that this path relies on persons' agreement (whether tacit, hypothetical, public reason, or voting processes) to be governed and to the actions, policies and decisions of a government, and for citizens' agreement to carry force, it must be based on conditions that meet the criteria of autonomy. And, again, our concern here is not an all-things-considered judgment about legitimacy, but about the ability of citizens to fulfill their responsibilities of legitimation. There are several ways in which the exploitation of data, psychological profiles, social media, and algorithmic systems creates a drag on legitimation. Consider first Christman's test for legitimacy, which requires that political principles be such that reasonable citizens would not be understandably alienated from them.67 The fake groups that the IRA created and advertised were 65 Sandra C. Matz et al., "Psychological Targeting as an Effective Approach to Digital Mass Persuasion" 114, no. 48 (2017): 12717. 66 Matz et al., 12717. 67 Christman, The Politics of Persons: Individual Autonomy and Socio-Historical Selves, 239. (Emphasis in original.) 33 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. promoted by fake user profiles, but eventually were followed by real people, referenced real issues, and mimicked names and language of genuine social, religious, and political groups (e.g., Stop All Immigrants, Blacktivist, LGBTQ United). The language, the titles, and linking to real issues and groups drew in people who came to express their views and wills on topics with real stakes. However, by joining, engaging with, and thereby amplifying those groups and messages (and in some cases populating in-person events68), it is likely that their wills were not autonomous because the expression is one from which people would be alienated. That is, if people were to "engage in sustained critical reflection," which requires reflecting on how they came to their commitment, the expressions would be incompatible with their sense of self and practical identity over time.69 And, if a person earnestly engaged in political debates about policing, rights of LGBTQ+ persons, and immigration were to reflect on the fact that Facebook groups like Stop All Immigrants, Blacktivist, and LGBTQ United were ersatz communities built by Russian agents to exploit and enrage them, they would in all likelihood be alienated from their support. In other words, the social media campaigns undermine the legitimation process by undermining the authenticity condition for autonomy. 68 Wylie recounts that in Summer 2014, Cambridge Analytica began creating fake Facebook and other social media pages and groups with politically charged identities that were tailored to be fed into the news feeds of susceptible users by recommendation algorithms. As people joined the groups, Cambridge Analytica would set up meetings in small spaces (in order to make the group feel big). Because of the nature of the groups, and the nature of the content fed into the groups, "[p]eople would show up and find a fellowship of anger and paranoia." Wylie, Mindf*ck: Cambridge Analytica and the Plot to Break America, 128. These groups would, when large enough and prompted by Cambridge Analytica, meld with similar groups, creating a network of engaged, angry people who felt they were under siege. Wylie, 127–29. 69 Christman, The Politics of Persons: Individual Autonomy and Socio-Historical Selves, 155. 34 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Consider next the ways in which psychological profiling and targeting on social media works. It works, first, to engage people by keeping them scrolling on social media feeds.70 It also seeks to create sustained engagement. One mechanism for creating such engagement is provoking strong emotions-including affinity for a cause and anger at opponents. The tactics draw people in and exploit their emotions. These tactics, for reasons we describe in chapter 5.2, are instances of the affective, deliberative, and social challenges to agency. They thus undermine the quality of people's agency and thereby conflict with their autonomy. This diminishes people's ability to perform their legitimating responsibilities. The fake accounts also promoted falsehoods. First, they promoted falsehoods about the nature of the groups themselves. But they also promulgated false claims. This circumvents persons' autonomy by undermining its epistemic requirements. And it creates yet another drag on the legitimation process. Finally, there is an issue that goes back to Feinberg's understanding of "ideal" autonomy and to social/relational conceptions of autonomy.71 People are not isolated, atomistic individuals. They are parts of families, communities, and social groups. Such groups are vital to autonomy both in the sense that they are important in causing people to flourish and to develop their autonomy and in the sense that autonomy involves the ability to act and realize one's values within communities. The tactics of Cambridge Analytica and the IRA undermine that. They specifically seek to create ersatz communities that displace genuine ones and to rend broader 70 Wylie, Mindf*ck: Cambridge Analytica and the Plot to Break America, 127; Nir Eyal, Hooked: How to Build Habit-Forming Products, ed. Ryan Hoover, 1st ed. (New York, New York: Portfolio, 2014). 71 See sections 2.2.2, 2.6, and 3.3. 35 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. communities into factions that neither trust nor communicate effectively with each other. Deep disputes are an important and ever-present facet of democratic polities. However, encouraging disparate groups to further distrust others and undermining the ability of those groups to communicate conflicts with the social facet of autonomy and, hence, on the legitimation process. 8.5. Once more past the pole The issue of legitimation in the cases of PredPol, Cambridge Analytica, and the IRA is momentous. How police exercise their power to put others in harm's way, efforts to affect an election via social profiling, and interference of a hostile nation in a U.S. presidential election cut to the quick of democratic legitimacy. But issues of political authority and legitimation are not limited to matters at the heart of democratic procedures and at the far reaches of state sanctioned exercise of power. The extend also to ordinary administrative tasks as well, among them risk assessment and teacher evaluation. COMPAS raises similar issues to PredPol on the normative authority arm of political legitimacy. It is no doubt true that efficient allocation of scarce resources for supervising people in the criminal justice system is good policy. There are substantial open questions about that system overall, of course. But if we accept the premises that the state should sanction some kinds of actions with a range of penalties and that supervision and resources (e.g., drug and alcohol treatment, job programs, counseling, housing) are appropriate responses, assessing persons for placement within those programs would appear to be a reasonable approach. But, as we explain in section 3.4.3, the use for which COMPAS is designed and suited is different than the use to which it is put in Loomis. Hence, even if though there is evidence that COMPAS is wellcalibrated in assessing risk, that same evidence does not underwrite use for sentencing purposes. 36 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. As with PredPol, there is a disconnect between evidence and tool use, and that creates a drag on the epistemic facet of legitimation via normative authority. The VAM cases are a bit simpler in that the tool itself has substantial flaws, as we discuss in sections 1.1 and 3.4. Evaluating teachers is a reasonable policy goal. However, the fact that the tools to do so have such big flaws preclude use of EVAAS and TVAAS from legitimation via normative authority. There is not a similar impediment to legitimation via democratic will. Note two things, though. First, that some institution, government, law, or policy is legitimate-which is to say based on a justified exercise of political power-is not the same as it being morally justified. Political legitimacy is in that sense a lower bar. Second, the fact that the algorithmic systems in these cases do not place a drag on the legitimation process is not a sufficient condition to ensure legitimacy. As we make clear in section 8.4, our concern is that some kinds of algorithmic systems hinder citizens' abilities to exercise their responsibility to legitimate policies. VAMs don't obviously hinder fulfilling that responsibility. 8.6. Conclusions Autonomy is foundational for political legitimacy. It grounds each of the accounts we canvas in section 8.2, including the hybrid account we endorse. We argued that one component part of legitimacy is the process of legitimation. That is, citizens have a responsibility to exercise their autonomy to legitimate governments, policies, laws, and actions. Algorithmic systems can impede citizens from fulfilling those responsibilities. PredPol (and to an extent COMPAS) is an example of an impediment to legitimation via normative authority. Whatever evidence there is of PredPol advancing a good policy goal (and there is some), that deflects from assessing how 37 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. PredPol is actually used. That conflicts with the access condition, whereby belief in a policy's value has to be linked to evidence for its actual value. The cases of Cambridge Analytica and the IRA undermine the second arm of political legitimacy. They create conditions where citizens would be alienated form the source of their beliefs and attitudes, they substitute ersatz involvement for genuine involvement, and generate false beliefs. But we should reiterate here that political legitimacy goes much further than the process of legitimation. For one, there are questions about different levels or targets of legitimacy. The fact, if it is, that policies using particular technologies create a drag on legitimation or, worse, are not legitimate all-things-considered, does not tell us whether the agencies using the technologies, the municipalities, states, or nations using those technologies are legitimate, or much else. It is important not to over-interpret drags on legitimation to make broader conclusions and legitimacy per se. Related is that legitimate authority can get things wrong and still retain legitimacy to exercise authority. That breadth of legitimacy is why our conclusions here are limited to the ability of citizens to exercise their responsibilities. References Bond-Graham, Darwin. "All Tomorrow's Crimes: The Future of Policing Looks a Lot like Good Branding." SF Weekly, October 30, 2013. https://www.sfweekly.com/news/alltomorrows-crimes-the-future-of-policing-looks-a-lot-like-good-branding/. Cadwalladr, Carole, and Emma Graham-Harrison. "Revealed: 50 Million Facebook Profiles Harvested for Cambridge Analytica in Major Data Breach." The Guardian, March 17, 2018, sec. News. 38 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Christman, John. The Politics of Persons: Individual Autonomy and Socio-Historical Selves. Reissue edition. Cambridge: Cambridge University Press, 2011. Confessore, Nicholas. "Cambridge Analytica and Facebook: The Scandal and the Fallout so Far." The New York Times, April 4, 2018, sec. U.S. Davies, Harry. "Ted Cruz Campaign Using Firm That Harvested Data on Millions of Unwitting Facebook Users." The Guardian, December 11, 2015, sec. US news. https://www.theguardian.com/us-news/2015/dec/11/senator-ted-cruz-president-campaignfacebook-user-data. Detrow, Scott. "What Did Cambridge Analytica Do during the 2016 Election?" NPR. Accessed June 17, 2020. https://www.npr.org/2018/03/20/595338116/what-did-cambridgeanalytica-do-during-the-2016-election. DiResta, Renee, Kris Shaffer, Becky Ruppel, David Sullivan, Robert Matney, Ryan Fox, Jonathan Albright, and Ben Johnson. "The Tactics & Tropes of the Internet Research Agency." U.S. Senate Documents, October 1, 2019. Edwards, Ezekiel. "Predictive Policing Software Is More Accurate at Predicting Policing than Predicting Crime." American Civil Liberties Union (blog). Accessed July 8, 2020. https://www.aclu.org/blog/criminal-law-reform/reforming-police/predictive-policingsoftware-more-accurate-predicting. Enoch, David. "Authority and Reason-Giving." Philosophy and Phenomenological Research 89, no. 2 (2014): 296–332. Estlund, David M. Democratic Authority: A Philosophical Framework. Princeton, N.J.: Princeton University Press, 2008. Eyal, Nir. Hooked: How to Build Habit-Forming Products. Edited by Ryan Hoover. 1st ed. New York, New York: Portfolio, 2014. Ferguson, Andrew Guthrie. The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement. New York: NYU Press, 2017. Goode, Erica. "Sending the Police before There's a Crime." The New York Times, August 15, 2011, sec. U.S. Grossman, Lev, Mark Thompson, Jeffrey Kluger, Alice Park, Bryan Walsh, Claire Suddath, Eric Dodds, et al. "The 50 Best Inventions." Time, November 28, 2011. Hakim, Danny, and Matthew Rosenberg. "Data Firm Tied to Trump Campaign Talked Business with Russians." The New York Times, March 17, 2018, sec. U.S. Hamburger, Tom. "Cruz Campaign Credits Psychological Data and Analytics for Its Rising Success." Washington Post, December 13, 2015, sec. Politics. Hoff, Sam. "Professor Helps Develop Predictive Policing by Using Trends to Predict, Prevent Crimes." The Daily Bruin, April 26, 2013. https://web.archive.org/web/20191216105955/https://dailybruin.com/2013/04/26/profess or-helps-develop-predictive-policing-by-using-trends-to-predict-prevent-crimes/. Howard, Philip N., Bence Kollanyi, Samantha Bradshaw, and Lisa-Maria Neudert. "Social Media, News and Political Information during the US Election: Was Polarizing Content Concentrated in Swing States?" Oxford Internet Institute, Project on Computational Propaganda, 2017. Hunt, Priscilla, Jessica Saunders, and John Hollywood. "Evaluation of the Shreveport Predictive Policing Experiment." Santa Monica, CA: RAND Corporation, 2014. https://www.rand.org/pubs/research_reports/RR531.html. 39 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. Illinois v. Wardlow, 528 U.S. 119 (2000). Kaiser, Brittany. Targeted: The Cambridge Analytica Whistleblower's inside Story of How Big Data, Trump, and Facebook Broke Democracy and How It Can Happen Again. Harper, 2019. Kang, Cecilia, and Sheera Frenkel. "Facebook Says Cambridge Analytica Harvested Data of up to 87 Million Users." The New York Times, April 4, 2018, sec. Technology. Mather, Kate, and Richard Winton. "LAPD Uses Its Helicopters to Stop Crimes before They Start." Los Angeles Times, March 7, 2015, sec. California. Matheson, Craig. "Weber and the Classification of Forms of Legitimacy." The British Journal of Sociology 38, no. 2 (1987): 199–215. Matz, Sandra C., Michal Kosinski, Gideon Nave, and David J. Stillwell. "Psychological Targeting as an Effective Approach to Digital Mass Persuasion" 114, no. 48 (2017): 12714–19. Meijer, Albert, and Martijn Wessels. "Predictive Policing: Review of Benefits and Drawbacks." International Journal of Public Administration 42, no. 12 (September 10, 2019): 1031– 39. Oshana, Marina. Personal Autonomy in Society. 1st ed. Aldershot, Hants, England ; Burlington, VT: Routledge, 2006. Peter, Fabienne. Democratic Legitimacy. New York: Routledge, 2008. ---. "The Epistemic Circumstances of Democracy." In The Epistemic Life of Groups: Essays in the Epistemology of Collectives, edited by Miranda Fricker and Michael Brady, 133– 149. Oxford: Oxford University Press, 2016. ---. "The Grounds of Political Legitimacy." Journal of the American Philosophical Association, forthcoming. Pettit, Philip. Just Freedom: A Moral Compass for a Complex World. 1st edition. New York: W. W. Norton & Company, 2014. Rawls, John. "Justice as Fairness: Political Not Metaphysical." Philosophy & Public Affairs 14, no. 3 (1985): 223–51. ---. Political Liberalism. New York: Columbia University Press, 2005. Raz, Joseph. The Morality of Freedom. Oxford: Oxford University Press, 1988. Rubel, Alan, Clinton Castro, and Adam Pham. "Algorithms, Agency, and Respect for Persons." Social Theory & Practice 43, no. 3 (July 2020): 547–72. Schwartz, Joanna C. "After Qualified Immunity." Columbia Law Review 120, no. 2 (2020): 309– 88. Select Committee on Intelligence, United States Senate. "Report of the Select Committee on Intelligence, United States Senate, on Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume I: Russian Efforts against Election Infrastructure with Additional Views." Washington, D.C: Select Committee on Intelligence, United States Senate, 2019. ---. "Report of the Select Committee on Intelligence, United States Senate, on Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume II: Russia's Use of Social Media and Additional Views." Washington, D.C: Select Committee on Intelligence, United States Senate, 2019. Shklar, Judith. "The Liberalism of Fear." In Liberalism and the Moral Life, edited by Nancy L. Rosenblum. Cambridge, MA: Harvard University Press, 1989. 40 This is a preprint. Please cite to revised, final version in Algorithms & Autonomy: The Ethics of Automated Decision Systems, forthcoming with Cambridge University Press, when available. U.S. Department of Justice. "Report on the Investigation into Russian Interference in the 2016 Presidential Election, Volume I ('Mueller Report')." Washington, D.C.: U.S. Department of Justice, March 2019. U.S. v. Internet Research Agency, LLC, No. 1:18-cr-00032 (U.S. District Court for the District of DC February 16, 2018). Waldman, Ari Ezra. "Power, Process, and Automated Decision-Making." Fordham Law Review 88, no. 2 (September 29, 2019): 613–32. Weaver, Matthew. "Facebook Scandal: I Am Being Used as Scapegoat – Academic Who Mined Data." The Guardian, March 21, 2018, sec. UK news. Weber, Max. Economy and Society: An Outline of Interpretive Sociology. University of California Press, 1978. Wylie, Christopher. Mindf*ck: Cambridge Analytica and the Plot to Break America. New York: Random House, 2019.