In the article ‘Testimonial injustice in medical machine learning’, Pozzi argues that the prescription drug monitoring programme (PDMP) leads to testimonial injustice as physicians are more inclined to trust the PDMP’s risk scores over the patient’s own account of their medication history.1 Pozzi further develops this argument by discussing how credibility shifts from patients to machine learning (ML) systems that are supposedly neutral. As a result, a sense of distrust is now formed between patients and physicians. While there are merits (...) to Pozzi’s main argument of epistemic injustice caused by PDMPs, Pozzi mentions but ultimately glosses over the problem of automation bias. In this commentary, I will discuss automation bias and the affect it has on clinical decision making as well as a technical problem exacerbated by the usage of PDMPs that can potentially cause physical harms. It is reiterated in the article that the confidence physicians have in the PDMP’s risk scores over the patient’s testimony leads to misplaced trust in the ML systems. What Pozzi describes here is known as automation bias, which occurs when there is an over-reliance on ML systems. …. (shrink)

State prescription drug monitoring programs (PDMPs) use proprietary, predictive software platforms that deploy algorithms to determine whether a patient is at risk for drug misuse, drug diversion, doctor shopping, or substance use disorder (SUD). Clinical overreliance on PDMP algorithm‐generated information and risk scores motivates clinicians to refuse to treat—or to inappropriately treat—vulnerable people based on actual, perceived, or past SUDs, chronic pain conditions, or other disabilities. This essay provides a framework for challenging PDMP algorithmic discrimination as disability discrimination under (...) federal antidiscrimination laws, including a new proposed rule interpreting section 1557 of the Affordable Care Act. (shrink)

Pozzi provides a thought-provoking account of how machine-learning clinical prediction models (such as Prediction Drug Monitoring Programmes (PDMPs)) may exacerbate testimonial injustice.1 In this response, I generalise Pozzi’s concerns about PDMPs to traditional models of clinical practice and question the claim that inaccurate clinicians are necessarily preferential to inaccurate machine-learning models. I then explore Pozzi’s concern that such models may deprive patients of a right to ‘convey information’. I suggest that machine-learning tools may be used to enhance, rather (...) than frustrate, this right, through the perspective of hermeneutical justice. Pozzi objects to the introduction of machine-learning risk prediction tools in clinical care, due to their being ‘epistemically opaque’, often inaccurate, and a threat to patients’ testimonial justice. Through the example of psychiatry, I suggest this stance idealises traditional models of clinical practice. I propose that clinicians’ judgements are often equally vulnerable to opaque subjectivity, bias and epistemic injustice. For instance, the use of clinical observation or collateral interviewing might be conceptualised as a threat to testimonial justice, by neglecting patients’ own perspectives or voices. In practice, clinical risk prediction often follows inconsistent and somewhat subjective clinical heuristics, which are difficult to summarise or evaluate. Further, although Pozzi diminishes machine-learning …. (shrink)

Pozzi1 has thoroughly analysed testimonial injustices in the automated Prediction Drug Monitoring Programmes (PDMPs) case. Although Pozzi1 suggests that ‘the shift from an interpersonal to a structural dimension … bears a significant moral component’, her topical investigation does not further conceptualise the type of collective knowledge practices necessary to achieve epistemic justice. As Pozzi1 concludes: ‘this paper shows the limitations of systems such as automated PDMPs, it does not provide possible solutions’. In this commentary, we propose that an (...) Ubuntu perspective—which, so far, has received little attention in connection with epistemic injustice in medical machine learning (MML)—can fill this gap. This perspective can be used to mitigate some of the harmful effects of epistemic injustices by encouraging community involvement and forward-looking responsibility. It may be instructive to note first and foremost that the debate on epistemic justice and artificial intelligence in medicine has mostly been restricted to the Western World, that is, Europe and North America. There has been very little insight or accommodation of insights from the Global South. Ubuntu is a concept originating from Southern Africa that focuses on the close connections and mutual …. (shrink)

This paper seeks to understand the treatment effect of Prescription Drug Monitoring Programs on opioid prescription rates. Using county-level panel data on all opioid prescriptions in the U.S. between 2006 and 2015, we investigate whether state interventions like PDMPs have heterogeneous treatment effects at the sub-state level, based on regional and temporal variations in policy design, extent of urbanization, race, and income. Our models comprehensively control for a set of county and time fixed effects, countyspecific and time-varying demographic controls, (...) potentially endogenous time-series trends in prescription rates, and other state-level opioid interventions such as Naloxone Access and Good Samaritan laws, Medicaid expansion, and the provision of Methadone Assistance Treatment. We find that PDMPs are only effective in reducing prescription rates if they obligate doctors to check for patients' history prior to filling out a prescription, but the frequency at which a state requires its PDMP to be updated is irrelevant to its effectiveness. Moreover, the significant treatment effects of PDMPs are almost exclusively driven by urban and predominantly white counties, with the relatively more affluent regions showing greater responsiveness than their less affluent counterparts. (shrink)

Machine learning (ML) systems play an increasingly relevant role in medicine and healthcare. As their applications move ever closer to patient care and cure in clinical settings, ethical concerns about the responsibility of their use come to the fore. I analyse an aspect of responsible ML use that bears not only an ethical but also a significant epistemic dimension. I focus on ML systems’ role in mediating patient–physician relations. I thereby consider how ML systems may silence patients’ voices and relativise (...) the credibility of their opinions, which undermines their overall credibility status without valid moral and epistemic justification. More specifically, I argue that withholding credibilitydue tohow ML systems operate can be particularly harmful to patients and, apart from adverse outcomes, qualifies as a form of testimonial injustice. I make my case for testimonial injustice in medical ML by considering ML systems currently used in the USA to predict patients’ risk of misusing opioids (automated Prediction Drug Monitoring Programmes, PDMPs for short). I argue that the locus of testimonial injustice in ML-mediated medical encounters is found in the fact that these systems are treated asmarkers of trustworthinesson which patients’ credibility is assessed. I further show how ML-based PDMPs exacerbate and further propagate social inequalities at the expense of vulnerable social groups. (shrink)

Parts I through III of this paper will examine several, increasingly comprehensive forms of aggregation, ranging from insurance reimbursement “lock-in” programs to PDMPs to completely unified electronic medical records. Each part will advocate for the adoption of these aggregation systems and provide suggestions for effective implementation in the fight against opioid misuse. All PDMPs are not made equal, however, and Part II will, therefore, focus on several elements — mandating prescriber usage, streamlining the user interface, ensuring timely data (...) uploads, creating a national data repository, mitigating privacy concerns, and training doctors on how to respond to perceived doctor-shopping — that can make these systems more effective. In each part, we will also discuss the privacy concerns of aggregating data, ranging from minimal to significant, and highlight the unique role of stigma in motivating these concerns. In Part IV, we will conclude by suggesting remedial steps to offset this loss of privacy and to combat the stigma around SUDs and mental health disorders in general. (shrink)

Artificial intelligence-based (AI) technologies such as machine learning (ML) systems are playing an increasingly relevant role in medicine and healthcare, bringing about novel ethical and epistemological issues that need to be timely addressed. Even though ethical questions connected to epistemic concerns have been at the center of the debate, it is going unnoticed how epistemic forms of injustice can be ML-induced, specifically in healthcare. I analyze the shortcomings of an ML system currently deployed in the USA to predict patients’ likelihood (...) of opioid addiction and misuse (PDMP algorithmic platforms). Drawing on this analysis, I aim to show that the wrong inflicted on epistemic agents involved in and affected by these systems’ decision-making processes can be captured through the lenses of Miranda Fricker’s account of hermeneutical injustice. I further argue that ML-induced hermeneutical injustice is particularly harmful due to what I define as an automated hermeneutical appropriation from the side of the ML system. The latter occurs if the ML system establishes meanings and shared hermeneutical resources without allowing for human oversight, impairing understanding and communication practices among stakeholders involved in medical decision-making. Furthermore and very much crucially, an automated hermeneutical appropriation can be recognized if physicians are strongly limited in their possibilities to safeguard patients from ML-induced hermeneutical injustice. Overall, my paper should expand the analysis of ethical issues raised by ML systems that are to be considered epistemic in nature, thus contributing to bridging the gap between these two dimensions in the ongoing debate. (shrink)

Some physicians, in their care of patients at risk of misusing opioids, use machine learning (ML)-based prediction drug monitoring programmes (PDMPs) to guide their decision making in the prescription of opioids. This can cause a conflict: a PDMP Score can indicate a patient is at a high risk of opioid abuse while a patient expressly reports oppositely. The prescriber is then left to balance the credibility and trust of the patient with the PDMP Score. Pozzi1 argues that a prescriber (...) who downgrades the credibility of a patient’s testimony based on a low PDMP Score is epistemically and morally unjustified and contributes to a form of testimonial injustice. This results in patients being silenced, excluded from decision-making processes and subjected to structural injustices. Additionally, the use of ML systems in medical practices raises concerns about perpetuating existing inequalities, overestimating their capabilities and displacing human authority. However, almost the very same critiques apply to human-based systems. Formalisation, ML systems included, should instead be viewed positively,2 and precisely as a powerful means to begin eroding these and other problems in ethically sensitive domains. In this case, the epistemic virtues of formalisation include promoting transparency, consistency and replicability in decision making. Rigorous ML systems can also help ensure that models …. (shrink)

Giorgia Pozzi’s feature article1 on the risks of testimonial injustice when using automated prediction drug monitoring programmes (PDMPs) turns the spotlight on a pressing and well-known clinical problem: physicians’ challenges to predict patient behaviour, so that treatment decisions can be made based on this information, despite any fallibility. Currently, as one possible way to improve prognostic assessments of patient behaviour, Machine Learning-driven clinical decision support systems (ML-CDSS) are being developed and deployed. To make her point, Pozzi discusses ML-CDSSs that (...) are supposed to deliver physicians an accurate estimation of the likelihood of narcotic, sedative and stimulant opioid misuse by a given patient (like, e.g., ‘NarxCare’). Regarding cases of deviating assessment between human evaluators and automated systems, the medico-ethical discussion has so far mainly focused on disagreement between clinicians and Machine Learning (ML) algorithms,2 for example in ‘second opinions’ that have been reconstructed as disagreements between physician testimony and ML-CDSS testimony.3 Examples such as those of PDMPs, by contrast, draw attention to situations in which patient testimony potentially contradicts ML-CDSS testimony. Pozzi examines decision-making situations in which ML-CDSS testimony factually overrides the testimony of the person affected. Thus, in the presented case of patient Kathryn, the testimony of her own physical and mental condition was insufficient to overturn the unfavourable outcome for …. (shrink)

In my paper entitled ‘Testimonial injustice in medical machine learning’,1 I argued that machine learning (ML)-based Prediction Drug Monitoring Programmes (PDMPs) could infringe on patients’ epistemic and moral standing inflicting a testimonial injustice.2 I am very grateful for all the comments the paper received, some of which expand on it while others take a more critical view. This response addresses two objections raised to my consideration of ML-induced testimonial injustice in order to clarify the position taken in the paper. (...) The first maintains that my critical stance toward ML-based PDMPs idealises standard medical practice. Moreover, it claims that the ML-induced testimonial injustice I discuss is not substantially different from situations in which it emerges in human–human interactions. The second claims that my analysis does not establish a link to issues of automation bias, even if these are to be considered the core of testimonial injustice in ML. In the following, I address each objection in turn. Gillett3 argues that my critical stance towards using risk prediction tools such as PDMPs implies the idealisation of standard (ie, non-ML-mediated) modes of clinical practice. Considering certain uses of ML in a different setting, that is, psychiatry, the author goes as far as claiming that ‘traditional models of clinical practice in psychiatry are far from a utopia, free from epistemic injustice, which Pozzi’s argument risks proposing’. Since this statement does not represent what I intend to suggest, I am glad to have the possibility to clarify …. (shrink)