AI in Healthcare: Development Phases, Ethical Questions, and Liability Risks

The following has been edited for length and clarity.

Q: You recently spoke at the TDC Group Executive Advisory Board meeting about legal and ethical issues relating to medical artificial intelligence (AI). Can you summarize your main themes?

A: While there are a lot of AI-related conversations going on regarding the risks of medical professional liability exposure for clinicians and organizations, the existing case law is pretty thin on the subject, and it may be that the liability risk is less than is supposed. At the same time, there are many ethical quandaries as well as sources of reputational risk to consider.

Overall, I am on the more optimistic side about medical AI. I think AI has real benefits for hospital systems, physicians, other practitioners, and ultimately patients. At the same time, there are serious concerns about privacy, bias, consent, and accountability, as I discuss in my work. And while it would be nice to say that keeping a human in the loop will always help, in fact the literature suggests that in many instances, adding human decision makers can produce performance worse than humans or AI acting alone.

Q: One thing you emphasized was that there are various goals for adopting medical AI, but they may have very different ethical valences. Can you say more?

A: I think the first question to ask is, “Why are we building this?” What are we hoping medical AI might do for us that is worth the risk or cost?

Here are some possible answers initially suggested by my friend and sometimes coauthor W. Nicholson Price:

• Democratizing expertise: Taking the expertise we already have in our healthcare system and making it more accessible, scaling up good care.
• Automating drudgery: Reducing the time physicians, nurses, and others are spending on tasks like billing, so they can do the tasks where they add the most value. We have seen healthcare organizations experiencing early success in freeing up clinician time and attention for dealing directly with patients, rather than email, billing, etc.
• Optimizing resources: Using AI to determine which patients should get priority for a particular healthcare treatment. We have seen a bit of this in the organ allocation space.
• Pushing frontiers: This is how Silicon Valley tends to talk, and this is what AI hype revolves around. In this category are applications seeking to assist an already very good dermatologist or radiologist, for example, in becoming even better.

The market is very focused on that last goal, while much of the actual implementation we are seeing relates to the second goal, but a lot of the ethical value is in the first goal, and we have to think about aligning incentives with ethics.

Q: During your talk, you discussed how legal and ethical issues show up at four different phases of AI development. Can you walk us through the four phases?

A: As I tell friends in computer science, when I describe the phases of AI development, it is a little like a poet telling you how to build a carburetor: beautiful, but insufficient to assist with a repair. Similarly, as a lawyer and ethicist, the way I break it down into phases is not so much a reflection on how these things are built and developed as an effort to show distinct legal and ethical issues that accompany each step of the process.

Phase 1: Acquiring the Data

Medical AI is data hungry. At the same time, we want to make sure it is the right data for the task to avoid dataset bias (one of several forms of bias to worry about in this space). Some good examples of this problem might be algorithms to detect cardiac risk that are trained on patient data only or primarily from men, and image identification systems for melanoma trained on only or primarily images of pale skin.

So right off the bat, there is a challenge in making sure we are able to create datasets that are diverse and robust, and exactly what that means has to be guided by the task we hope to tackle.

But assembling diverse and robust datasets might, in some instances, be in tension with patient privacy. Do we need to explicitly ask patients for permission to use their data to train medical AI? Does it depend on how difficult the data is to reidentify? For example, would it be enough to strip the HIPAA identifiers from the data? Is actual permission needed, or could notice without seeking permission be sufficient? If we are thinking that the way we will do this is by boilerplate language buried in a front-door consent for treatment, it is worth asking whether the game is worth a candle. We may be better off relying on other technologies of governance, as we might call them, rather than individual consent. This might include creating governance boards: Patient representatives appropriately mirroring the patients whose data is being used would get to deliberate on whether to approve use in a particular case.

Phase 2: Building and Validating the Model

How do you know when a model is good enough to be used on real patients? There are tensions between protecting intellectual property (thus commercial opportunities) and promoting trustworthiness with patients and civil society.

There are interesting and hard questions about how much validation should be done by regulators, a hospital system, or third-party assessors. Larger healthcare systems may have an easier (though far from easy!) time conducting a thorough internal vetting process. Smaller healthcare systems might piggyback on large systems’ knowledge or form collaborative networks, but that is easier said than done.

Phase 3: Test the Model in Real-World Settings

So now you are ready to use the model on real patients. There are lots of hard questions, including ones related to informed consent. What, if anything, do you have to tell patients before this AI is used in their care? I have written extensively about the legal and ethical obligations as to AI and informed consent, but I think this is a very tricky space.

One way into it is by analogy. If a substitute surgeon scrubbed in and did your surgery, and the doctor did not tell you, that would be a clear violation of obligations as to informed consent. On the other hand, when a pediatrician decides whether to give antibiotics to a child who presents with something that might be bacterial or viral, their thought process is a kind of black box. It might swirl together back issues of JAMA, memories from medical school, collegial conversations, similar patients, etc. We don’t think patients are entitled to be given a list of every input that led to our physician’s recommendation.

Which is the right analogy to medical AI? The answer might depend on questions like: How much does the physician know about the AI, and what reasons does the physician have for thinking the AI is trustworthy enough to further patient care? Has the FDA reviewed the AI, and in what way? What about internal review and approval of the AI by the hospital system? Does the physician have reason to believe this is information that is particularly important to the patient in making a decision? How serious of a decision is it for the patient? In thinking of what we owe a patient, all these might be relevant.

Liability, which I know you will ask me about in a moment, is also a big piece of the considerations in this third phase.

Also, the hospital system or physician’s office that is implementing the AI will want to try to assess what some call contextual bias: How well will the AI (trained to work in potentially a different practice setting or for different patients) perform when used on patients in this hospital system or practice? Properly doing that assessment is a big undertaking.

Phase 4: Broad Dissemination

Say you’ve got a model that’s successful, and it’s working for your patient population. Now the question is, who else can it help?

And what obligations, ethically speaking, do you have to make sure it’s reaching the users it can benefit, even if many of them are in resource-poor settings?

Q: Issues of liability are front and center for many of our members. What is the current thinking about how AI might increase, decrease, or change liability for practitioners?

A: Thanks. At the risk of sounding even more like a lawyer, I’ll start by saying, “It is complicated!” Many people have a hand in decisions about AI that will affect patients’ welfare: designers, regulators, hospital purchasers, insurers, and medical practitioners. Changes at one level, including changes to rules about liability, will interact with decision making at other levels—but trying to determine how in advance is very challenging.

With so much changing so quickly and so many kinds of medical AI, anticipating liability issues is also challenging. That said, overall, I think physicians, other clinicians, and healthcare entities may be more protected than they think: Existing case law does not bear out the degree of worry I’ve heard some express about AI liability risks for healthcare professionals.

In earlier work I expressed the concern that fears of liability might result in too little medical AI adoption. Since tort law typically privileges the standard of care—regardless of its effectiveness in a particular case—the worry is that healthcare practitioners might be tempted to disregard the AI when it recommends something other than the standard of care in precisely the cases where the AI is doing better, thus helping to achieve some of the promise of personalization. While it is possible there may come a time when failing to use the AI itself becomes an instance of falling below the standard of care, the history of malpractice allegations related to other new technologies suggests that the medico-legal system is fairly conservative in penalizing failure to adopt new medical technologies unless and until there is very widespread adoption.

On the flip side, because of the complexity of these cases, the difficulties in proving liability, and the complications of trade secrets, not to mention more generally understanding how the underlying AI works and explaining it to the trier of fact, I suspect that medical AI cases are not particularly attractive cases for plaintiffs’ lawyers to bring. Outside of aggregate litigation such as class actions, which might face their own challenges, lawsuits pertaining to medical AI seem less attractive than more typical medical malpractice lawsuits from the plaintiff’s bar perspective, given the significant litigation costs.

Some more recent work by my friend (and sometimes coauthor) Michelle Mello and her colleague Neel Guhahas discussed what we see in the small number of cases in this space that have resulted in published decisions. They find the courts interposing several obstacles to liability, including:

• Showing that the acceptance of or the departure from the AI recommendation was unreasonable.
• Securing an expert witness who could specify why something was unreasonable.
• Proving foreseeability that the model’s output was inappropriate for a particular patient.
• Demonstrating causation. They have to show that had the model been better and made a different recommendation, not only would the physician have followed it, but a different care outcome would have occurred.
• Accessing enough information about the algorithm's design and operation to write their complaint in the first place and to survive a motion to dismiss.

Q: Your talk also addressed the risk of bias in medical AI. What should we look out for?

A: Before I get there, I want to mention that it is easy to understand everything I am saying as being a bit of a “downer” about medical AI. We might have a litany of legitimate complaints about AI for healthcare: Your AI gives poor explanations and is not very candid about the reasons why it came to a certain conclusion. Your AI is leading to errors in your care. And your AI is quite biased. For each, though, the important point to realize is: These concerns could also apply to your healthcare practitioner. The real question is: Under what circumstances can the AI or a particular combination of AI and human beings working together improve on these dimensions?

When it comes to bias specifically, it is important to understand key forms of bias relevant to AI—including those that start with us.

Practitioners’ bias can be translated into the dataset used to train the AI. For instance, women typically receive less intensive cholesterol management treatment than men, despite being more likely to suffer certain cardiac events. In healthcare, longstanding biased habits like these can become incorporated into new technologies, contributing to what’s known as measurement classification bias.

Bias is a persistent challenge, and one for which there is no silver bullet. Our best defense is for developers to start thinking about bias the moment they start building, rather than to ask questions on the back end.

AI-powered tools can help practitioners, and can help them deliver better care to more patients. We should not allow Perfect to become the enemy of Good.