Optimism about our ability to enhance societal decision‐making by leaning on Machine Learning (ML) for cheap, accurate predictions has palled in recent years, as these ‘cheap’ predictions have come at significant social cost, contributing to systematic harms suffered by already disadvantaged populations. But what precisely goes wrong when ML goes wrong? We argue that, as well as more obvious concerns about the downstream effects of ML‐based decision‐making, there can be moral grounds for the criticism of these predictions themselves. We introduce (...) and defend a theory of predictive justice, according to which differential model performance for systematically disadvantaged groups can be grounds for moral criticism of the model, independently of its downstream effects. As well as helping resolve some urgent disputes around algorithmic fairness, this theory points the way to a novel dimension of epistemic ethics, related to the recently discussed category of doxastic wrong. (shrink)

Racial capitalism depends on the reproduction of an existing racialized economic order. In this article, I argue that the disavowal of past injustice is a central way in which this reproduction is ensured and that market-based forms of knowledge production, such as for-profit predictive practices, play a crucial role in facilitating this disavowal. Recent debates about the fairness of algorithms, data justice, and predictive policing have intensified long-standing controversies, both popular and academic, about the way in which statistical and financial (...) modes of accounting and predicting articulate, represent and produce ascriptive categories of hierarchically ordered social difference, and reproduce unjust social hierarchies and inequalities. These debates have productively problematized the racial lives of seemingly apolitical predictive technologies and demanded the re-politicization of predictive practices. What has been missing from these debates so far, however, is a more explicit engagement with ways in which anti-racist movements and activists themselves have contested the entanglements of prediction and race making. I turn to a recent prominent example, namely the contestation over racial discrepancies in subprime lending to examine how fair lending activists have conceptualized and troubled the reproduction of a racialized economic order through for-profit predictive practices in the decade before the Great Financial Crisis. I situate this particular example in the broader historical and political context of politicizing prediction that first emerged with the ascendancy of a liberal, individualist-proprietary conception of risk, and the political problem space to which this has given rise. My analysis shows that actuarial conceptions of fairness continue to reverberate in anti-racist contestations of for-profit predictive practices, and that they tend to marginalize and undercut more radical strands of critique of the racialization of financial markets. Insofar as these modalities of contestation implicitly reproduce a liberal, proprietary-individualist conception of risk, I argue, they fail to effectively challenge the quasi-alchemical transformation of injustice into personal responsibility, and thus contribute to the disavowal of past injustice. (shrink)

This special theme of Big Data & Society explores connections, relationships, and tensions that coalesce around data, power, and racial formation. This collection of articles and commentaries builds upon scholarly observations of data substantiating and transforming racial hierarchies. Contributors consider how racial projects intersect with interlocking systems of oppression across concerns of class, coloniality, dis/ability, gendered difference, and sexuality across contexts and jurisdictions. In doing so, this special issue illuminates how data can both reinforce and challenge colorblind ideologies as well (...) as how data might be mobilized in support of anti-racist movements. (shrink)

Efforts to promote equitable public policy with algorithms appear to be fundamentally constrained by the “impossibility of fairness” (an incompatibility between mathematical definitions of fairness). This technical limitation raises a central question about algorithmic fairness: How can computer scientists and policymakers support equitable policy reforms with algorithms? In this article, I argue that promoting justice with algorithms requires reforming the methodology of algorithmic fairness. First, I diagnose the problems of the current methodology for algorithmic fairness, which I call “formal algorithmic (...) fairness.” Because formal algorithmic fairness restricts analysis to isolated decision-making procedures, it leads to the impossibility of fairness and to models that exacerbate oppression despite appearing “fair.” Second, I draw on theories of substantive equality from law and philosophy to propose an alternative methodology, which I call “substantive algorithmic fairness.” Because substantive algorithmic fairness takes a more expansive scope of analysis, it enables an escape from the impossibility of fairness and provides a rigorous guide for alleviating injustice with algorithms. In sum, substantive algorithmic fairness presents a new direction for algorithmic fairness: away from formal mathematical models of “fair” decision-making and toward substantive evaluations of whether and how algorithms can promote justice in practice. (shrink)

In this paper I argue that some examples of what we label ‘algorithmic bias’ would be better understood as cases of institutional bias. Even when individual algorithms appear unobjectionable, they may produce biased outcomes given the way that they are embedded in the background structure of our social world. Therefore, the problematic outcomes associated with the use of algorithmic systems cannot be understood or accounted for without a kind of structural account. Understanding algorithmic bias as institutional bias in particular (as (...) opposed to other structural accounts) has at least two important upshots. First, I argue that the existence of bias that is intrinsic to certain institutions (whether algorithmic or not) suggests that at least in some cases, the algorithms now substituting as pieces of institutional norms or rules are not “fixable” in the relevant sense, because the institutions they help make up are not fixable. Second, I argue that in other cases, changing the algorithms being used within our institutions (rather than getting rid of them entirely) is essential to changing the background structural conditions of our society. (shrink)

Responsibility has become a central concept in AI ethics; however, little research has been conducted into practitioners’ personal understandings of responsibility in the context of AI, including how responsibility should be defined and who is responsible when something goes wrong. In this article, we present findings from a 2020–2021 data set of interviews with AI practitioners and tech workers at a single multinational technology company and interpret them through the lens of feminist political thought. We reimagine responsibility in the context (...) of AI development and deployment as the product of work cultures that enable tech workers to be responsive and answerable for their products over the long and short term. From our interviews, we identify three key pain points in understanding the distribution of responsibility between actors and developing responsible design and deployment practices: (1) unstable business ecosystems and AI lifecycles, which require an approach to responsibility that accounts for the dynamic nature of these systems; (2) the issue of incentivizing engineers to take responsibility for the mundane maintenance practices essential to the functioning of AI systems and (3) the need to overcome individual and structural barriers to taking ownership over AI products and their effects. From these findings, we make three recommendations based on feminist theory: (1) organisations should move from a static model of responsibility to a dynamic and ethically motivated response-ability; (2) companies need to revalue care and maintenance practices; and (3) firms must move away from individualistic ideas of responsibility towards fostering wider cultures of responsibility. (shrink)

This paper clarifies why bias cannot be completely mitigated in Machine Learning (ML) and proposes an end-to-end methodology to translate the ethical principle of justice and fairness into the practice of ML development as an ongoing agreement with stakeholders. The pro-ethical iterative process presented in the paper aims to challenge asymmetric power dynamics in the fairness decision making within ML design and support ML development teams to identify, mitigate and monitor bias at each step of ML systems development. The process (...) also provides guidance on how to explain the always imperfect trade-offs in terms of bias to users. (shrink)

Artificial intelligence tools used in employment decision-making cut across the multiple stages of job advertisements, shortlisting, interviews and hiring, and actual and potential bias can arise in each of these stages. One major challenge is to mitigate AI bias and promote fairness in opaque AI systems. This paper argues that the equal opportunity merit principle is an ethical approach for fair AI employment decision-making. Further, explainable AI can mitigate the opacity problem by placing greater emphasis on enhancing the understanding of (...) reasonable users and affected persons as to the AI output. Both the equal opportunity merit principle and explainable AI should be integrated in the design and implementation of AI employment decision-making systems so as to ensure, as far as possible, that the AI output is arrived at through a fair process. (shrink)

Algorithms are used to calculate and govern varying aspects of public life for efficient use of the vast data available about citizens. Assuming that algorithms are neutral and efficient in data-based decision making, algorithms are used in areas such as criminal justice and welfare. This has ramifications on the ideal of democratic self-government as algorithmic decisions are made without democratic deliberation, scrutiny or justification. In the book Democracy without Shortcuts, Cristina Lafont argued against “shortcutting” democratic self-government. Lafont’s critique of shortcuts (...) turns to problematise taken-for-granted practices in democracies that bypass citizen inclusion and equality in authoring decisions governing public life. In this article, I extend Lafont’s argument to another shortcut: the algocratic shortcut. The democratic harms attributable to the algocratic shortcut include diminishing the role of voice in politics and reducing opportunities for civic engagement. In this article, I define the algocratic shortcut and discuss the democratic harms of this shortcut, its relation to other shortcuts to democracy and the limitations of using shortcuts to remedy algocratic harms. Finally, I reflect on remedy through “aspirational deliberation”. (shrink)