A Framework for Assurance Audits of Algorithmic Systems

Proceedings of the 2024 Acm Conference on Fairness, Accountability, and Transparency (forthcoming)
  Copy   BIBTEX

Abstract

An increasing number of regulations propose the notion of ‘AI audits’ as an enforcement mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently have little to no agreed upon practices, procedures, taxonomies, and standards. We propose the ‘criterion audit’ as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations’ ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit, and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which ‘bias audits’ for hiring algorithms can be performed, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion as informed by experiences in performing these audits in practice highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of such methodology.

Links

PhilArchive

External links

  • This entry has no external links. Add one.
Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

Civil society and social auditing.Adrian Henriques - 2001 - Business Ethics, the Environment and Responsibility 10 (1):40–44.
Civil society and social auditing.Adrian Henriques - 2001 - Business Ethics: A European Review 10 (1):40-44.
Contemporary Auditing: Issues and Cases.Michael Chris Knapp - 1999 - South Western Educational Publishing.
Auditing the audits: are we missing the point?Harish V. Kurup, Andy McMurtrie & Steve Sarasin - 2007 - Journal of Evaluation in Clinical Practice 13 (5):829-830.

Analytics

Added to PP
2024-04-02

Downloads
128 (#143,457)

6 months
128 (#30,687)

Historical graph of downloads
How can I increase my downloads?

Author Profiles

Benjamin Lange
Ludwig Maximilians Universität, München
Borhane Blili-Hamelin
Columbia University (PhD)
Jovana Davidovic
University of Iowa
1 more

Citations of this work

No citations found.

Add more citations