Skip to main content

Advertisement

SpringerLink
Log in

Social context of the issue of discriminatory algorithmic decision-making systems

  • OPEN FORUM
  • Published:
AI & SOCIETY Aims and scope Submit manuscript

Abstract

Algorithmic decision-making systems have the potential to amplify existing discriminatory patterns and negatively affect perceptions of justice in society. There is a need for a revision of mechanisms to address discrimination in light of the unique challenges presented by these systems, which are not easily auditable or explainable. Research efforts to bring fairness to ADM solutions should be viewed as a matter of justice and trust among actors should be ensured through technology design. Ideas that move us to explore the notions of justice within the field of political thinking, aiming to identify the elements that frame the social context of the discriminatory decisions produced by algorithmic decision-making solutions. Our explorations suggest that the efforts to bring fairness to ADMs should be seen as a matter of justice and that trust in these systems can be ensured through careful technology design.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data availability

Not applicable.

Notes

  1. There is no certainty that Calicles was an actual parson. It is thought that he was a fictional character invented by Plato to embody all the injustices and opposition committed against Socrates.

  2. First book of social contract page 8.

  3. Fourth book of social contract page 107.

  4. Manish Tewari is the former Union Minister of the Government of India, is also a lawyer and member of parliament.

  5. Part One: The Problem of Knowledge, sections four and five The Liberal Conception of Justice and Communicating Justice: The Second-Order Problem of Knowledge, respectively.

  6. Mills, Charles W. (1997). The Racial Contract. Cornell University Press Ithaca and London. pp. 3–4.

  7. Mills, Charles W. (1997). The Racial Contract. Cornell University Press Ithaca and London. pp. 1–2, 5, and 7.

  8. According to Haslanger there exists multiple modalities of feminism including but not exclusive to cultural feminism, liberal feminism, radical feminism, ecofeminism, anarcho-feminism, feminism of difference, gender feminism, equality feminism, Marxist feminism, socialist feminism, separatist feminism, philosophical feminism, Islamic feminism, and lesbian feminism.

  9. Retardment of the community and its consequences like the racism appreciation of rape and homosexuality, for example. 17.

  10. For the BLM movement USA is a State of Violence against the Black community because of the following facts: the majority of inmate population of the country are Black, the systematic racist police assaults against Black community members is founded in historical prejudice, Black homosexuals and transgenders are targeted by undervaluing fetichism from the heteropatriarchal generalized and protected standard of living, around 500, 000 undocumented immigrants are Black, several Darwinian experiments have targeted Black women, and other elements exposed by Garza in “A Herstory of the #BlackLivesMatter Movement.”.

  11. Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) software is a case management and decision support tool developed and owned by Northpointe (now Equivant) that is used by U.S. courts to predict the likelihood of a defendant becoming a recidivist.

  12. The “high-risk care management” program helps to provide chronically ill people with extended care at hospitals, access to specially trained nursing staff and allocate extra primary-care visits for closer monitoring.

  13. Dastin’s study points that Amazon does not publish gender breakdown since 2017, which is the same year the company stopped a project that used AI to suggest the top five candidates to be hired out of a pool of resumes.

References

  • Ajunwa I (2021) The auditing imperative for automated hiring. Harv. J.L. & Tech.. Retrieved from https://ssrn.com/abstract=3437631 or http://dx.doi.org/https://doi.org/10.2139/ssrn.3437631

  • Amorós C (1990) El feminismo: Senda no transitada de la Ilustración. Isegoria 1:139–150

    Google Scholar 

  • Aquinas T (1993) Selected philosophical writings. Oxford University Press

    Google Scholar 

  • Aquinas T (2023) Summa theologiae II–II. Benziger Brothers

  • Barnett RE (2000) The structure of liberty: justice and the rule of law. Oxford University Press, Oxford

    Book  Google Scholar 

  • Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G (2014) Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff 33:1123–1131. https://doi.org/10.1377/hlthaff.2014.0041pmid:25006137

    Article  Google Scholar 

  • Black LM (2021) Retrieved from https://blacklivesmatter.com/about/

  • Bornstein S (2019) Antidiscriminatory algorithms. Alabama Law Rev 70:519–572

    Google Scholar 

  • Carson EA (2015) Prisoners in 2014. Bureau of Justice Statistics. Retrieved from: http://www.bjs.gov/index.cfm?ty=pbdetail&iid=5387

  • Chambers K (2018) Augustine on justice: a reconsideration of city of God, book 19. Polit Theory 19(5):382–396. https://doi.org/10.1080/1462317X.2018.1438781

    Article  Google Scholar 

  • Clements J (2008) Confucius: a biography. Sutton Publishing

  • Cohen PN (1999) Book review: the social contract. Rev Radical Polit Econ 31(2):102–105. https://doi.org/10.1177/048661349903100208

    Article  Google Scholar 

  • Confucius (2003) Confucius: analects—with selections from traditional commentaries. Hackett Publishing, Translated by E. Slingerland

    Google Scholar 

  • Constitution Center (2021) Amendment XV. Retrieved from https://constitutioncenter.org/interactive-constitution/amendment/amendment-xv

  • Constitution Center (2021) Amendment XIV. Retrieved from https://constitutioncenter.org/interactive-constitution/amendment/amendment-xiv

  • Coontz S (2011) A strange stirring: the feminine mystique and American women at the dawn of the 1960s. Basic Books

    Google Scholar 

  • Copleston F (1999) A history of philosophy 3. Continuum International Publishing Group

  • Daniels N (1989) Reading rawls: critical studies on rawls’ “a theory of justice.” Stanford University

    Book  Google Scholar 

  • Dastin J (2018) Amazon scraps secret AI recruiting tool that showed bias against women. Reuters October 10th, 2018, withdrawn July 30th, 2021

  • Department of Economic and Social Affairs. (2013). Inequality matters: Report of the world social situation 2013. United Nations.

  • Dodds ER (1959) Plato gorgias. A revised text with introduction and commentary. Oxford University Press, Oxford

    Google Scholar 

  • Dred S (1857) Missouri state archives: Missouri’s case, 1846–1857. Missouri Digital Heritage. Retrieved from https://www.sos.mo.gov/archives/resources/africanamerican/scott/scott.asp. July 2021.

  • Dressel J, Farid H (2018) The accuracy, fairness, and limits of predicting recidivism. Sci Adv. https://doi.org/10.1126/sciadv.aao5580

    Article  Google Scholar 

  • Duggan L, Hunter ND (1995) Sex wars: Sexual dissent and political culture. Routledge

  • Eijo de Tezanos PMA (2016) Are Hispanics discriminated against in the US criminal justice system? Graduate Research Posters. Poster 12. https://scholarscompass.vcu.edu/gradposters/12

  • Engerman SL, Sokoloff KL, Urquiola M, Acemoglu D (2002) Factor endowments, inequality, and paths of development among new world economies. Economía 3(1):41–109

    Article  Google Scholar 

  • Fjeld J, Achten N, Hilligoss H, Nagy A, Srikumar M (2020) Principled artificial intelligence: mapping consensus in ethical and rights-based approaches to principles for AI. Berkman Klein Center for Internet & Society, Harvard Law School

    Google Scholar 

  • Flores AW, Lowenkamp CT, Bechtel K (2023) False positives, false negatives, and false analyses: A rejoinder to “machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks.” Federal Probation 80(2)

  • Freedman EB (2003) No turning back: the history of feminism and the future of women, 464. Ballantine Books

    Google Scholar 

  • Friedan B (1964) The feminine mystique. Norton, W.W

    Google Scholar 

  • Garza A (2023) A herstory of the #BlackLivesMatter movement. The Feminist Wire

  • Gillis S, Howie G, Munford R (eds) (2007) Third wave feminism: a critical exploration. Palgrave Macmillan

  • Green M (2008) Hobbes on justice. Soc Polit Philos 33

  • Haslanger S, Tuana N, O'Connor P (2012) In: Zalta EN (Ed) Topics in feminism. The Stanford Encyclopedia of Philosophy

  • Hileman G, Steele S (2016) Accuracy of claims-based risk scoring models. Society of Actuaries

  • Hobbes T (1980) In: Moya C, Escotado A (eds) Leviatán. Editora Nacional

  • Isbister J (2001) Capitalism and justice: envisioning social and economic fairness. Lynne Rienner Publishers

  • Jenofonte (1993) Recuerdos de Sócrates; económico; banquete; apología de Sócrates. Gredos

  • Johnston I (2002) Lecture on Machiavelli's the prince. Malaspina University College.

  • Kehl D, Guo P, Kessler S (2017) Algorithms in the criminal justice system: assessing the use of risk assessments in sentencing. responsive communities initiative. Berkman Klein Center for Internet & Society, Harvard Law School

  • Kelly P (2013) The politics book. Dorling Kindersley

  • Locke J (1690) Two treatises of government: an essay concerning the true original, extent, and end of civil government. Black Swan

  • Machan TR (1975) Law, justice, and natural rights. Western Ontario Law Rev 14:119–130

    Google Scholar 

  • Machiavelli N (1990) Allocution made to a magistrate. Polit Theory 18(4):525–527. https://doi.org/10.1177/0090591790018004002

    Article  Google Scholar 

  • Machiavelli N (1531) The discourses. Translated by Leslie, J. & Walker, S.J., revisions by Brian Richardson (2003). Penguin Books

  • Marx K, Engels F (1848) The manifesto of the Communist Party

  • McGuire DL (2010) At the dark end of the street: Black women, rape, and resistance—A new history of the civil rights movement from Rosa Parks to the rise of black power. Random House

    Google Scholar 

  • Mills CW (1997) The racial contract. Cornell University Press

    Google Scholar 

  • National Archives (2021) African American heritage. Retrieved August 2021, from https://www.archives.gov/research/african-americans/black-power

  • Nozick R (1974) Anarchy, state and utopia. Basic Books

    Google Scholar 

  • Obermeyer Z, Powers B, Vogeli C, Mullainathan S (2019) Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464):447–453. https://doi.org/10.1126/science.aax2342

    Article  Google Scholar 

  • Okin SM (1989) Justice, gender, and the family. Basic Books

    Google Scholar 

  • Olsthoorn J (2015) Hobbes on justice, property rights and self-ownership. Hist Polit Thought 36(3):471–498 http://www.jstor.org/stable/26228629

  • Parel A (1990) Machiavelli's notions of justice: text and analysis. Polit Theory 18(4):528–544. Sage Publications. http://www.jstor.org/stable/191540

  • Paterman C (1988) The sexual contract. Polity Press

    Google Scholar 

  • Patrick C (2015) Aquinas on law and justice, conflict of human law and justice in the orderly society. Southern New Hampshire University

  • Platón (1999) Diálogos. Obra completa. Volumen VIII: Leyes (Libros I-VI). introducción, traducción y notas de Francisco Lisi. Gredos

  • Platón (2003a) Diálogos. Obra completa. Gredos

  • Platón (2003a) Diálogos. Obra completa en 9 volúmenes. Volumen IV: La República. Gredos

  • Platón. (2003b) Diálogos. Obra completa en 9 volúmenes. Volumen V: Parménides. Teeteto. Sofista. Político. Gredos

  • Powell J (1996) John Locke: natural rights to life, liberty, and property. In: The freeman. The Foundation for Economic Education

  • Rawls J (1971) A theory of justice. Harvard University Press, Harvard

    Book  Google Scholar 

  • Rawls J (1993) Political liberalism. Columbia University Press, Columbia

    Google Scholar 

  • Rawls J (1999) The law of peoples. Harvard University Press, Harvard

    Google Scholar 

  • Rawls J (2001) Justice as fairness: a restatement. Harvard University Press, Harvard

    Book  Google Scholar 

  • Reisman G (2019) Classical economics versus the exploitation theory. Mises Daily Articles. Mises Institute. Retrieved from https://mises.org/library/classical-economics-vs-exploitation-theory. July 2021.

  • Riley J (1989) Justice under capitalism. Nomos 31:122–162. https://www.jstor.org/stable/24219468

  • Rousseau JJ (1959) OEuvres complètes. Gallimard, Bibliothèque de la Pléiade

    Google Scholar 

  • Rousseau JJ (1762) Contrato social. Espasa-Calpe

  • Schall JV (1998) At the limits of political philosophy. CUA Press

  • Taylor U (1998) The historical evolution of black feminist theory and praxis. J Black Stud 29(2):234–253

    Article  Google Scholar 

  • Tewari M (2019) Oh justice! Raisina debates. Observer Research Foundation ORF blog Retrieved from https://www.orfonline.org/expert-speak/justice-nazi-germany-54934/. July 2021.

  • The Economic Commission for Latin America and the Caribbean (ECLAC) (2021) Afrodescendants and the matrix of social inequality in Latin America: challenges for inclusion. Summary. United Nations

  • Tong R (2009) Feminist thought: a more comprehensive introduction (3rd edition). Westview Press (Perseus Books), 289:284–285

  • UN Women (2019) Progress of the world’s women 2019–2020: Families in a changing world. United Nations. Retrieved from progress.unwomen.org

  • UN (2023) Department of Economic and Social Affairs. Sustainable Development. Retrieved from sdgs.un.org/goals

  • Varona D, Suarez JL (2022a) Discrimination, bias, fairness, and trustworthy AI. Appl Sci 12(12):5826. https://doi.org/10.3390/app12125826

    Article  Google Scholar 

  • Varona D, Lizama-Mue Y, Suarez JL (2021) Machine learning’s limitations in avoiding automation of bias. AI Soc 30:197–203

    Article  Google Scholar 

  • Varona D, Suarez JL (2022b) Analysis of the principled-AI framework’s constraints in becoming a methodological reference for trustworthy-AI design. In: Engel U, Quan-Haase A, Xun Liu S, Lyberg LE (eds) Handbook of computational social science, vol 1. Routledge Taylor and Francis Group: Oxfordshire, ISBN 9780367456528

  • Varona D, Suarez JL (2023) Principled AI engineering challenges towards trust-worthy AI. Ethics Inf Technol

  • Vogeli C, Shields AE, Lee TA, Gibson TB, Marder WD, Weiss KB, Blumenthal D (2007) Multiple chronic conditions: prevalence, health consequences, and implications for quality, care management, and costs. J Gen Intern Med 22(suppl. 3):391–395. https://doi.org/10.1007/s11606-007-0322-1pmid:18026807

    Article  Google Scholar 

  • Walzer M (1983) Spheres of justice: a defense of pluralism and equality. Basic Books

  • Wolff RP (1977) Understanding Rawls: a reconstruction and critique of “a theory of justice.” Princeton University Press

    Google Scholar 

  • Yu D (2009) Confucio para el alma o las claves milenarias para ser feliz. Editorial Planeta

Download references

Author information

Authors and Affiliations

  1. CulturePlex Laboratory, University College 2210, London, ON, N6A 3K7, Canada

    Daniel Varona & Juan Luis Suarez

Authors
  1. Daniel Varona
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan Luis Suarez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Varona.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Varona, D., Suarez, J.L. Social context of the issue of discriminatory algorithmic decision-making systems. AI & Soc (2023). https://doi.org/10.1007/s00146-023-01741-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00146-023-01741-x

Keywords

Search

Navigation