What Are the Challenges to Responsible Use of AI in Medicine?

Medicine has been at the forefront of efforts to translate recent advances in Artificial Intelligence (AI) into concrete benefits for individuals and social institutions. The high-stakes nature of clinical interventions, however, underscores the importance of ensuring that well-intended efforts to improve medical practices and the delivery of patient care actually translate into better health outcomes and more efficient delivery of equitable care. This panel explores challenges for responsible innovation using AI in medicine and concrete approaches for overcoming some of these obstacles in practice.

Expert Panel

Alex John London is a philosopher who will outline some of the common socio-technical challenges of using AI in healthcare. This talk will introduce the audience to how AI systems work and examine some of the complexities of assembling the various elements necessary to produce an effective, efficient and reliable AI system. It is commonly argued that explainability plays a critical role in this process but the presenter will argue that it is neither necessary nor sufficient for the responsible development and deployment of AI in medicine. Instead, it is argued that we need better practices of specifying the intended use case or indication for AI systems and then generating real-world evidence of safety and efficacy before AI systems are deployed. The goal of this talk is to help the audience see parallels between AI systems and complex pharmaceutical interventions and to place proper emphasis on the role of rigorous prospective evaluation of AI systems in medicine.

Maarten van Smeden is a medical statistician and epidemiologist who will illustrate with concrete cases how some of the problems outlined in general terms by the first presenter materialize in practice. These cases illustrate inefficiencies in current practices surrounding AI development including massive duplication of effort and the paucity of transferable learning from these efforts. This portion of the talk drives home, with concrete examples, challenges to using AI to accomplish the goals of making health systems more effective and efficient. This talk will also illustrate challenges to implementing prediction models in practice, highlighting social as well as technical challenges to using AI to improve the delivery of healthcare.

Melissa McCradden is a bioethicist who will elaborate a practical proposal for identifying methodological considerations for the testing of AI systems. In particular, she explains how to adapt the requirements of clinical equipoise to AI systems in order to establish that clinical equipoise obtains. She will then explain which practices for system validation and evaluation can generate sufficiently reliable evidence to “disturb” or eliminate equipoise while also protecting the rights and interests of patient-participants. The goal of this talk is to establish that it is both practically feasible and ethically desirable to generate better, prospective, real-world evidence about the performance of AI systems.

Bringing Ideas Together

Although these three talks are distinct, they present a unified perspective on unique challenges facing the development and deployment of AI systems in medicine, shortcomings of current requirements for the responsible use of such systems and concrete steps that might be taken to advance the goals of creating more equitable, effective and efficient healthcare delivery. They show that popular concepts from the discourse on AI ethics may not be suited to the medical context (e.g., explainability) and demonstrate how to adapt established concepts in bioethics (such as clinical equipoise) and frameworks for governance from responsible AI to meet some of these challenges.