Abstract
In this paper we develop a general theory of algorithmic fairness. Drawing on Johnson King and Babic’s work on moral encroachment, on Gary Becker’s work on labor market discrimination, and on Strawson’s idea of resentment and indignation as responses to violations of the demand for goodwill toward oneself and others, we locate attitudes to fairness in an agent’s utility function. In particular, we first argue that fairness is a matter of a decision-maker’s relative concern for the plight of people from different groups, rather than of the outcomes produced for different groups. We then show how an agent’s preferences, including in particular their attitudes to error, give rise to their decision thresholds. Tying these points together, we argue that the agent’s relative degrees of concern for different groups manifest in a difference in decision thresholds applied to these groups.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
See Babic et al. (2021c) for an overview of AI/ML risks and Lander and Nelson (2021), for a recent call for fundamental legislative reform on regulating AI/ML from White House advisors on science affairs. Some of that reform is already underway. The European Union’s General Data Protection Regulation (2016) purports to provide a right to have automated decision making not based on “data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade-union membership” (Council Regulation 2016/679, 2016 O.J. (L 119) 51) although the scope of such a right is a matter of debate (Gerke et al., 2020) Meanwhile, Canada’s Bill C-27 would require an organization to provide an explanation of the prediction, recommendation or decision produced by its algorithmic system, including the “principal reasons or factors that led to the prediction.” Bill C-27, 44th Parliament, 1st Sess. (Canada 2022).
See e.g. Schwartz (2019). This kind of case corresponds to the now famous example of St. George’s Hospital Medical School in London in the 1980 s. While such glaring discrimination—taking points off for non-Caucasian names—is less likely with modern algorithms, giving less weight to certain groups in more subtle ways still occurs, as we will later discuss in much more detail.
For example, it has been documented that graduates of historically black colleges, such as Howard University, may be getting on average disadvantageous interest rates on their loans (Arnold, 2016).
It is worth noting that in this case Justices Thurgood Marshall and William Brennan wrote a dissenting opinion arguing that the degree and foreseeability of a disparate impact can be more probative of discriminatory intent than the Court recognizes. But here even the dissenting Justices acknowledge that discriminatory attitudes are the primary ingredient, so to speak, and that while impact may be relevant it is so insofar as it indicates these underlying attitudes (with the disagreement focusing on what kinds of facts can provide evidence of discriminatory attitudes—in the Strawsonian terms that we will introduce shortly, the Justices disagree about the relative merits of evidence of malice and evidence of indifference. We discuss these issues further in Sect. 3.1.
It should be noted that even if an attribution of attitudes is metaphorical, this does not mean that it is not sensible or useful. Indeed, the legal notion of corporate personhood requires that we impute a similar sort of fictitious mental content to corporate entities; as Chief Justice Marshall states in a well-known case before the US Supreme Court, “The great object of an incorporation is to bestow the character and properties of individuality on a collective and changing body.” Providence Bank v. Billings 29 U.S. 514 (1830). Still, it would be good to have a clear idea of what exactly the metaphor amounts to. We do not currently have such a thing.
The above is not intended as an exhaustive survey of extant approaches to thinking about algorithmic fairness. For example, Nabi and Shpitser (2018) and Nabi et al. (2019) develop a causal approach to algorithmic fairness: on their view, bias can be thought of in terms of the presence of an effect of a sensitive feature like race on the prediction along certain causal pathways. Meanwhile, we develop a theory that is grounded in Bayesian rational choice. For someone who wants to view bias from the perspective of economic rationality, and who who is critical of causal learning models—due, for example, to the difficulty in estimating independence and identifying causal models, or to our aforementioned point that causal models reflect “bias” in the sense in which a coin is biased but not in the sense in which a human decision-maker is biased—we have an alternative. However, for someone who is a proponent of causal inference and wishes to capture fairness from this perspective, Nabi and Shpitser (2018) and Nabi et al. (2019) have an answer. As to the bigger question of when and whether we should use causal inference models for predictive inference, that is beyond the scope of this project.
For a more complete history of the practice of redlining and its insidious effects, see Winling and Michney (2021).
This phenomenon is an instance of Simpson’s paradox, which describes situations where a target property that exists at a group level may not exist to the same extent when the group is subdivided on the basis of a feature that is not independent of the target property.
Someone might argue that we ought to identify bias with disparate impact on the grounds that the latter is readily measurable and so the theoretical identification would be useful for courts. However, if this is the sole reason offered for identification, then the case for identification is extremely weak. One might just as well argue that we ought to identify bias with sandwiches since it is very easy to tell when a sandwich is present. We hold that epistemology should not drive metaphysics in this fashion; at a minimum, for this kind of theoretical identification to be plausible one would have to argue that the hard-to-observe phenomena and the easy-to-observe phenomena are relevantly similar, such that the identification has some plausibility at the conceptual level. And now metaphysics returns to the scene. We hold that the identification has little conceptual plausibility, for reasons described below.
Strawson’s paper is dense and has been subject to much exegetical work. For an influential interpretation, to which we are congenial, see Watson (2004), especially p. 221.
For more on the difference between absolute and relative evaluation of agents’ degrees of concern, see Johnson King (2020).
The literature on human agency and responsibility has long distinguished between varieties of responsibility in a way that underpins the point we are making here (see Shoemaker, 2011, 2015; and cf. Watson, 1996). Using the language of this literature, we would put the point by saying that algorithms are neither accountable nor answerable for their decisions, but those decisions are nonetheless taken to be attributable to them.
Indeed, the revised Canadian bill on data protection, cited in footnote 1, requires certain algorithmic decisions to be accompanied by the “reasons or principal factors” that led to the decision.
The idea that we respond to agents’ decisions as evidence of what matters to the agent is also very familiar from the literature on moral responsibility in humans. For example, Shoemaker writes that “if something matters to me, it is just obvious that I regard it as having some sort of evaluative significance... These attitudes... reflect on me, on my deep self, and in particular on who I am as an agent in the world” (Shoemaker, 2011, p. 611). And Smith writes that even involuntary responses can “provide an important indication of a person’s underlying moral commitments, of who he is, morally speaking” (Smith, 2005, pp. 241–42). Smith emphasizes that these underlying moral commitments are “not necessarily consciously-held propositional beliefs, but rather tendencies to regard certain things as having evaluative significance. They comprise the things we care about or regard as important or significant” (Smith, 2005, p. 251). This is exactly how we propose to think about algorithmic bias: algorithms do not have anything like consciously-held propositional beliefs about the relative moral status of individuals or groups, but they certainly do have tendencies to regard certain things as having evaluative significance, as we will see shortly.
Washington v. Davis 426 U.S. 229 (1976). See also, Personnel Administrator of Massachusetts v Feeney, 442 U.S. 256 (1979) (noting that laws violating the Foureenth Amendment’s Equal Protection Clause are passed because of, not merely in spite of, their adverse effects upon an identifiable group).
For example, Griggs v Duke Power Co. 401 U.S. 424 (1971), interprets Title VII of the Civil Rights Act of 1964 to prohibit employment practices with a racially disparate impact.
This problem was first addressed in the late 1960 s and early 1970 s, including by Kaplan (1968), and Tribe (1971). Then in the 1980 s, including by Cohen (1981), Nesson (1985), and Thomson (1986). And more recently, with Colyvan et al. (2001), Schauer (2003), Redmayne (2008), Buchak (2014), and Cheng (2013). Most recently, philosophers such as Basu (2019) and Moss (2018) have written on this issue as well.
Recall also Smith’s point that evaluative judgments need not be propositional attitudes and may simply be “tendencies to regard certain things as having evaluative significance”; whatever exactly utilities might be, it seems clear that they are, at a minimum, ways of regarding certain things as having evaluative significance.
To make inferences about \(\theta _{\text {real}}\) on the basis of \(\theta _{\text {pseudo}}\) we would have to estimate the proportion of high aptitude minority group lawyers who fail to succeed, and the proportion of low aptitude minority group lawyers who succeed. Babic et al. (2021) propose a method for making this estimate for binary data.
One might think that if \(\text {E}[\hat{\theta }_A] > \text {E}[\hat{\theta }_B]\) then it must also be true that \(\overline{\theta }_A > \overline{\theta }_B\), since \(\text {E}[\hat{\theta }] = \theta\). But that is not necessarily so, because \(\text {E}[\hat{\theta }] = \theta\) only if the variances between \(\theta _A\) and \(\theta _B\) are the same. In the example we are currently considering, variances may not be identical. For instance, we can quite easily imagine a situation where the minority group B is on average disadvantaged—due to, say, socioeconomic and educational barriers—while containing many extraordinary unrecognized superstars, while group A, exploiting all of their advantage in a fairly uniform manner, is densely concentrated around a point of mediocrity.
\(s(p, I_{X_i})\) should be continuous, and s(p, 1) should be monotonically decreasing while s(p, 0) should be monotonically increasing.
In this project, we articulate \(p^*\) in terms of where s(p, 1) intersects with s(p, 0)—i.e., the point where there is no accuracy uncertainty, hence the point of zero epistemic risk. More generally, \(p^*\) is the minimum of the formal epistemic risk function, as articulated in Babic (2019).
Conversational models of responsibility are popular in the post-Strawson literature—see, for instance, McKenna (2012). The basic idea is that a blamer calls upon the blamee to either justify their action (in this case by showing that it is, contrary to appearances, supported by fairness considerations) or apologize and make amends.
The question of how to approach what would happen to those who are denied a loan is an interesting censored data problem, but we set it aside here.
Robert (2007) use this expression, but what is meant should be clear from the relationship between \(\alpha\), \(\beta\) and t in our model, as we explain.
Similarly, Corbett-Davies et al. (2017) argue that if one takes, as a persuasive conception of fairness, certain of the definitions that exist in the current literature—such as predictive disparity—then one can likewise express fairness constrained optimal decision making in terms of a threshold. This is not surprising given our discussion so far. Adding a constraint is equivalent to changing the utility function. The rational Bayesian continues to decide based on a threshold, but the particular value of that threshold changes. However, these authors do not say anything about which particular fairness constraint is normatively defensible, or why. This is our project.
See, for instance, Justice Powell’s infamous argument against quotas in Regents of the University of California v. Bakke 438 U.S. 265 (1978), and more recently Grutter v. Bollinger, 539 U.S. 306 (2003) (affirming the unconstitutionality of racial quotas).
Students for Fair Admissions, Inc. v. President and Fellows of Harvard College Docket 20-1199.
References
Aigner, D. J., & Cain, G. G. (1977). Statistical theories of discrimination in labor markets. Industrial and Labor Relations Review, 30(2), 175–187.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica.
Arnold, C. (2016). Graduates of historically black colleges may be paying more for loans: Watchdog group. NPR.
Arpaly, N. (2000). On acting against one’s best judgment. Ethics, 110(3), 488–513.
Arpaly, N., & Schroeder, T. (2013). In praise of desire. Oxford University Press.
Arrow, K. J. (1972). Models of job discrimination. In A. H. Pascal (Ed.), Racial discrimination in economic life (pp. 83–102). Lexington: Lexington Books, D. C. Heath and Co.
Arrow, K. J. (1972). Some mathematical models of race in the labor market. In A. H. Pascal (Ed.), Racial discrimination in economic life (pp. 187–2042). Lexington: Lexington Books, D. C. Heath and Co.
Arrow, K. J. (1974). The theory of discrimination. In O. Ashenfelter & A. Rees (Eds.), Discrimination in labor markets (pp. 1–33). Princeton University Press.
Autor, D. H. (2003). Lecture note: The economics of discrimination.
Babic, B. (2019). A theory of epistemic risk. Philosophy of Science, 86(3), 522–550.
Babic, B., Gaba, A., Tsetlin, I., & Winkler, R. L. (2021). Normativity, epistemic rationality, and noisy statistical evidence. British Journal for the Philosophy of Science.
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021). Beware explanations from AI in health care. Science, 373(6552), 284–286.
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021). Direct-to-consumer medical machine learning and artificial intelligence applications. Nature Machine Intelligence, 3, 283–287.
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021c). When machine learning goes off the rails. Harvard Business Review.
Basu, R. (2019). The wrongs of racist beliefs. Philosophical Studies, 9(176), 2497–2515.
Basu, R., & Schroeder, M. (2019). Doxastic wrongings. In B. Kim & M. McGrath (Eds.), Pragmatic encroachment in epistemology (pp. 181–205). Routledge.
Becker, G. S. (1957). The economics of discrimination (1st ed.). Chicago: University of Chicago Press.
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020). The state of artificial intelligence-based fda-approved medical devices and algorithms: An online database. Nature Digital Medicine 3.
Buchak, L. (2014). Belief, credence, and norms. Philosophical Studies, 169(2), 285–311.
Cheng, E. K. (2013). Reconceptualizing the burden of proof. Yale Law Journal, 122(5), 1254–1279.
Chohlas-Wood, A. (2020). Understanding risk assessment instruments in criminal justice. The Brookings Institution.
Cohen, J. (1981). Subjective probability and the paradox of the gatecrasher. Arizona State Law Journal, 1981(2), 627–634.
Colyvan, M., Regan, H. M., & Ferson, S. (2001). Is it a crime to belong to a reference class? Journal of Political Philosophy, 9(2), 168–181.
Corbett-Davies, S., & Goel, S. (2018). The measure and mismeasure of fairness: A critical review of fair machine learning.
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017). Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017).
Flores, A., Bechtel, K., & Lowenkamp, C. (2016). False positives, false negatives, and false analyses: A rejoinder to machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks. Federal Probation, 80(2), 38.
Foldessy, E. P. (1992). Largest metropolitan areas. Wall Street Journal.
Furlough, C., Stokes, T., & Gillan, D. J. (2021). Attributing blame to robots: The influence of robot autonomy. Human Factors, 63(4), 592–602.
Gendler, T. S. (2011). On the epistemic costs of implicit bias. Philosophical Studies, 1, 33–63.
Gerke, S., Minssen, T., & Cohen, I. (2020). Ethical and legal challenges of artificial intelligence-driven healthcare. Artificial intelligence in healthcare (pp. 295–336). Academic Press.
Gordon, C. (2021). The rise of AI in the transportation and logistics industry. Forbes.
Hall, J. (2019). How artificial intelligence is transforming digital marketing. Forbes.
Hellman, D. (2020). Measuring algorithmic fairness. Virginia Law Review, 106(4), 811–866.
Hidalgo, C., Orghian, D., Canals, J., Almeida, F., & Martin, N. (2021). How Humans Judge Machines. Cambridge: MIT Press.
Johnson, G. (2023). Are algorithms value free. Journal of Moral Philosophy (Forthcoming).
Johnson King, Z. (2020). Don’t know, don’t care? Philosophical Studies, 177(2), 413–431.
Johnson King, Z., & Babic, B. (2020). Moral obligation and epistemic risk. In M. Timmons (Ed.), Oxford studies in normative ethics (Vol. 10, pp. 81–105).
Kaplan, J. (1968). Decision theory and the factfinding process. Stanford Law Review, 20(6), 1065–1092.
Kleinberg, J., Ludwig, J., Mullainathan, S., & Rambachan, A. (2018). Algorithmic fairness. AEA Papers and Proceedings, 108, 22–27.
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. Proceedings of Innovations in Theoretical Computer Science (ITCS).
Lander, E., & Nelson, A. (2021). Americans need a bill of rights for an AI-powered world. WIRED.
Laplace, P. (1786). Sur les Naissances, les Mariages et les Morts à Paris Depuis 1771 Jusqu’à 1784 et Dans Toute L’étendue de la France, Pendant les Années 1781 et 1782. Mémoires de l’Académie Royale des Sciences Présentés par Diverse Savans.
Lima, G., Grgić-Hlača, N., & Cha, M. (2021). Human perceptions on moral responsibility of AI: A case study in AI-assisted bail decision-making. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, (CHI’21) (pp. 1–17).
Lima, G., Grgić-Hlača, N., Cha, M. (2023). Blaming Humans and Machines: What Shapes People’s Reactions to Algorithmic Harm. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, (CHI’23) (Vol. 372, pp. 1–26).
Liptak, A. (2017). Sent to prison by a software program’s secret algorithms. New York Times.
McKenna, M. (2012). Conversation and responsibility. Oxford University Press.
Moss, S. (2018). Probabilistic knowledge. Oxford University Press.
Nabi, R., Malinsky, D., & Shpitser, I. (2019). Learning optimal fair policies. In Proceedings of the 36th International Conference on Machine Learning, (ICML 36).
Nabi, R., & Shpitser, I. (2018). Fair inference on outcomes. In Proceedings of the Thirty Second AAAI Conference on Artificial Intelligence, (AAAI’18) (Vol. 32, pp. 1931–1940).
Nagel, T. (1976). Moral luck. Proceedings of the Aristotelian Society, Supplementary Volumes, 50, 137–155.
Nesson, C. (1985). The evidence or the event? On judicial proof and the acceptability of verdicts. Harvard Law Review, 98(7), 1357–1392.
Phelps, E. S. (1972). The statistical theory of racism and sexism. The American Economic Review, 62(4), 659–661.
Redmayne, M. (2008). Exploring the proof paradoxes. Legal Theory, 14(4), 281–309.
Rimol, M. (2021). Gartner forecasts worldwide artificial intelligence software market to reach \$62 billion in 2022. Gartner.
Robert, C. P. (2007). The Bayesian choice: From decision theoretic foundations to computational implementation. Springer.
Scanlon, T. (1998). What we owe to each other. Belknap Press.
Schauer, F. (2003). Profiles, probabilities, and stereotypes. Cambridge: Harvard University Press.
Schwartz, O. (2019). Untold history of AI: Algorithmic bias was born in the 1980s. IEEE Spectrum.
Shoemaker, D. (2011). Attributability, answerability, and accountability: Toward a wider theory of moral responsibility. Ethics, 121, 602–632.
Shoemaker, D. (2015). Responsibility from the margins. Oxford University Press.
Simoiu, C., Corbett-Davies, S., & Goel, S. (2017). The Problem of Infra-Marginality in Outcome Tests For Discrimination. The Annals of Applied Statistics, 11(3), 1193–1216.
Smith, A. (2005). Responsibility for attitudes: Activity and passivity in mental life. Ethics, 115(2), 236–271.
Spence, A. M. (1973). Job market signaling. Quarterly Journal of Economics, 87(3), 355–374.
Spence, A. M. (1974). Market signaling: Informational transfer in hiring and related screening processes. Cambridge: Harvard University Press.
Strawson, P. F. (1982). Freedom and resentment. In G. Watson (Ed.), Free will (1st ed., pp. 59–80). Oxford University Press.
Thomson, J. J. (1986). Liability and individualized evidence. Law & Contemporary Problems, 49(3), 199–219.
Tribe, L. H. (1971). Trial by mathematics: Precision and ritual in the legal process. Harvard Law Review, 84(6), 1329–1393.
Veloso, M., Balch, T., Borrajo, D., Reddy, P., & Shah, S. (2021). Artificial intelligence research in finance: Discussion and examples. Oxford Review of Economic Policy, 37(3), 564–584.
Verma, S., & Rubin, J. (2018). Fairness definitions explained. In Y. Brun, B. Johnson, A. Meliou (Eds.), Proceedings of the International Workshop on Software Fairness (pp. 1–7). ACM.
Watson, G. (1996). Two faces of responsibility. Philosophical Topics, 24, 227–248.
Watson, G. (2004). Responsibility and the limits of evil. Agency and answerability: Selected essays (pp. 219–259). Oxford University Press.
Williams, B. (1981). Moral luck. Cambridge: Cambridge University Press.
Winling, L. C., & Michney, T. M. (2021). The roots of redlining: Academic, governmental, and professional networks in the making of the new deal lending regime. Journal of American History, 108, 42–69.
Yong, E. (2018). A popular algorithm is no better at predicting crimes than random people. The Atlantic.
Funding
Boris Babic was funded by the INSEAD Desmerais fund for research in AI, by the Social Sciences and Humanities Research Council of Canada, and by the Schwartz Reisman Institute for Technology and Society.
Author information
Authors and Affiliations
Contributions
Both authors contributed equally to the writing of this paper at all stages of development.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Babic, B., Johnson King, Z. Algorithmic fairness and resentment. Philos Stud (2023). https://doi.org/10.1007/s11098-023-02006-5
Accepted:
Published:
DOI: https://doi.org/10.1007/s11098-023-02006-5