Abstract
In recent years, increasingly advanced artificial intelligence (AI), and in particular machine learning, has shown great promise as a tool in various healthcare contexts. Yet as machine learning in medicine has become more useful and more widely adopted, concerns have arisen about the “black-box” nature of some of these AI models, or the inability to understand—and explain—the inner workings of the technology. Some critics argue that AI algorithms must be explainable to be responsibly used in the clinical encounter, while supporters of AI dismiss the importance of explainability and instead highlight the many benefits the application of this technology could have for medicine. However, this dichotomy fails to consider the particular ways in which machine learning technologies mediate relations in the clinical encounter, and in doing so, makes explainability more of a problem than it actually is. We argue that postphenomenology is a highly useful theoretical lens through which to examine black-box AI, because it helps us better understand the particular mediating effects this type of technology brings to clinical encounters and moves beyond the explainability stalemate. Using a postphenomenological approach, we argue that explainability is more of a concern for physicians than it is for patients, and that a lack of explainability does not introduce a novel concern to the physician–patient encounter. Explainability is just one feature of technological mediation and need not be the central concern on which the use of black-box AI hinges.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In this paper, we focus our analysis mainly on physician use of black-box AI, as they most often deploy AI models and use particular models for diagnosis and/or treatment. Yet we recognize that other clinicians may also grapple with the use of AI in their work, and thus our analysis can be applied to healthcare workers in other roles, as well.
While we agree with much of Kiran’s analysis, it is worth questioning whether this aspect of his conclusion is justified. If technology completely frames and constrains what we see, like Kiran says, are we free to choose the extent to which we are constrained? It would distract from our thesis to address this question in full here, but we worry that Kiran’s suggestion does not take seriously the extent to which technology determines our gaze.
This can lead to widespread concerns about bias and discrimination, discussed further below.
Even though patients do not often request a robust explanation of the medical technologies routinely used in their care, there remains a normative question of whether they should, especially given certain features, such as potential algorithmic bias. While this is an interesting question worthy of further study, inquiring into the individual responsibilities of patients is beyond the scope of this paper.
This also raises legal and ethical concerns of accountability and liability at the level of the healthcare system. Who is to blame when an algorithm turns out to be flawed or systematically prescribes harmful treatments: the individual physician? The hospital? The firm that designed the AI model? (Grote & Berens, 2020).
While Ihde also introduces the epistemological magnification-reduction effects of technological mediation, we will focus here on Kiran’s three dimensions of technological mediation (enabling-constraining, revealing-concealing, involving-alienating), as these sufficiently build from Ihde and include elements of magnification-reduction throughout. In general, black box AI magnifies some parameters and necessarily reduces others, but the specific data points being magnified and reduced are context-specific and thus warrant unique analysis for individual AI algorithms.
This also highlights the multistability of black-box AI, or the various ways in which different users can engage with the technology. What feels very important to physicians may not actually be important to patients, and vice versa.
References
Aho, K. (2018). Existential medicine: Essays on health and illness. Rowman & Littlefield International.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Bishop, J. P. (2011). The anticipatory corpse: Medicine, power, and the care of the dying. University of Notre Dame Press.
Brouillette, M. (2017). Deep learning is a black box, but health care won’t mind. MIT Technology Review. https://www.technologyreview.com/2017/04/27/242905/deep-learning-is-a-black-box-but-health-care-wont-mind/
Campolo, A., Sanfilippo, M., Whittaker, M., & Crawford, M. (2017). AI Now 2017 Report. AI Now Institute. https://ainowinstitute.org/AI_Now_2017_Report.pdf
Clarke, A. 1973. Profiles of the future: An inquiry into the limits of the possible. Popular Library.
Gadamer, H.-G. (1996). The enigma of health: The art of healing in a scientific age. Stanford University Press.
Gertz, N. (2018). Nihilism and technology. Rowman & Littlefield.
Goodman, K. W. (2007). Ethical and legal issues in decision support. In E. S. Berner (Ed.), Clinical decision support systems: Theory and Practice (2nd ed., pp. 126–39). Springer.
Grote, T. & Berens, P. (2020). On the ethics of algorithmic decision-making in healthcare. Journal of Medical Ethics, 46, 205–211.
Hadler, R. A., Clapp, J. T., Chung, J. J., Gutsche, J. T., & Fleisher, L. A. (2021). Escalation and withdrawal of treatment for patients on extracorporeal membrane oxygenation (ECMO): A qualitative study. Annals of Surgery. https://doi.org/10.1097/SLA.0000000000004838
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., et al. (2019). Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25, 65–69. https://doi.org/10.1038/s41591-018-0268-3
Hawley, K. (2015). Trust and distrust between patient and doctor. Journal of Evaluation in Clinical Practice, 21(5), 798–801.
Heidegger, M. (1977). The question concerning technology, and other essays. Harper and Row.
Heidegger, M., & Boss, M. (2001). Zollikon seminars: Protocols, conversations, letters. Northwestern University Press.
Hofmann, B., & Svenaeus, F. (2018). How medical technologies shape the experience of illness. Life Sciences, Society and Policy, 14, 3.
Holzinger, A., Carrington, A., & Müller, H. (2020). Measuring the quality of explanations: The system causability scale (SCS). KI - Künstliche Intelligenz, 34(2), 193–198. https://doi.org/10.1007/s13218-020-00636-z
Ihde, D. (1977). Experimental phenomenology: An introduction. Putnam.
Ihde, D. (1990). Technology and the lifeworld: From garden to earth. Indiana University Press.
Ihde, D. (2002). Bodies in technology. University of Minnesota Press.
Kiran, A. H. (2015). Four dimensions of technological mediation. In R. Rosenberger & P.-P. Verbeek (Eds.), Postphenomenological investigations: Essays on Human-Technology Relations (pp. 123–140). Lexington Book.
Lipton, C. Z. (2016). The mythos of model interpretability. arXiv preprint. https://arxiv.org/pdf/1606.03490.pdf.
London, A.J. (2018). Groundhog day for medical artificial intelligence. Hastings Center Report, 48(3).
London, A. J. (2019). Artificial intelligence and black-box medical decisions: Accuracy versus explainability. Hastings Center Report, 49(1), 15–21.
Malone, J. (2019). Invasive medical technology: A postphenomenological variational analysis (Publication No. 13881260) [Doctoral Dissertation, Saint Louis University]. ProQuest Dissertations Publishing.
McDougall, R. J. (2019). Computer knows best? The need for value-flexibility in medical AI. Journal of Medical Ethics, 45(3), 156–160.
Mukherjee, S. (2017). A.I. versus M.D. The New Yorker. https://www.newyorker.com/magazine/2017/04/03/ai-versus-md
Nemati, S., Holder, A., Razmi, F., et al. (2018). An interpretable machine learning model for accurate prediction of sepsis in the ICU. Critical Care Medicine, 46, 547–553. https://doi.org/10.1097/CCM.0000000000002936
O’Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown.
Oudshoorn, N. E. J. (2011). Telecare technologies and the transformation of healthcare. Palgrave Macmillan.
Rajkomar, A., Oren, E., Chen, K., et al. (2018). Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1, 18. https://doi.org/10.1038/s41746-018-0029-1
Rosenberger, R., & Verbeek, P.-P. (2015). Postphenomenological investigations: Essays on human-technology relations. Lexington Books.
Simonite, T. (2018). Google's AI guru wants computers to think more like brains. Wired. www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/
Szegedy, C., Vanhoucke, V., Ioffe, S., et al. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, Nevada, 26 June–1 July 2016. Piscataway, NJ, pp. 2818–26.
Topol, E. J. (2019). Deep medicine: How artificial intelligence can make healthcare human again. Basic Books.
Verbeek, P.-P. (2005). What things do: Philosophical reflections on Technology, Agency, and Design. Pennsylvania State University Press.
Wang, F., Kaushal, R., & Khullar, D. (2020). Should health care demand interpretable artificial intelligence or accept “black box” medicine? Annals of Internal Medicine, 172, 59–60. https://doi.org/10.7326/M19-2548
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Friedrich, A.B., Mason, J. & Malone, J.R. Rethinking explainability: toward a postphenomenology of black-box artificial intelligence in medicine. Ethics Inf Technol 24, 8 (2022). https://doi.org/10.1007/s10676-022-09631-4
Accepted:
Published:
DOI: https://doi.org/10.1007/s10676-022-09631-4